Add doc page with best practices and other usage tips

docs/Readme.md

@@ -26,6 +26,7 @@ Running:
 
 Odds and ends:
 * [Frequently Asked Questions (FAQ)](faq.md#top)
+* [Best practices and other tips](usage-tips.md#top)
 * [CMake integration](cmake-integration.md#top)
 * [CI and other miscellaneous pieces](ci-and-misc.md#top)
 * [Known limitations](limitations.md#top)

docs/usage-tips.md

@@ -0,0 +1,95 @@
+<a id="top"></a>
+# Best practices and other tips on using Catch2
+
+## Running tests
+
+Your tests should be run in a manner roughly equivalent with:
+
+```
+./tests --order rand --warn NoAssertions
+```
+
+Notice that all the tests are run in a large batch, their relative order
+is randomized, and that you ask Catch2 to fail test whose leaf-path
+does not contain an assertion.
+
+The reason I recommend running all your tests in the same process is that
+this exposes your tests to interference from their runs. This can be both
+positive interference, where the changes in global state from previous
+test allow later tests to pass, but also negative interference, where
+changes in global state from previous test causes later tests to fail.
+
+In my experience, interference, especially destructive interference,
+usually comes from errors in the code under test, rather than the tests
+themselves. This means that by allowing interference to happen, our tests
+can find these issues. Obviously, to shake out interference coming from
+different orderings of tests, the test order also need to be shuffled
+between runs.
+
+However, running all tests in a single batch eventually becomes impractical
+as they will take too long to run, and you will want to run your tests
+in parallel.
+
+
+<a id="parallel-tests"></a>
+## Running tests in parallel
+
+There are multiple ways of running tests in parallel, with various level
+of structure. If you are using CMake and CTest, then we provide a helper
+function [`catch_discover_tests`](cmake-integration.md#automatic-test-registration)
+that registers each Catch2 `TEST_CASE` as a single CTest test, which
+is then run in a separate process. This is an easy way to set up parallel
+tests if you are already using CMake & CTest to run your tests, but you
+will lose the advantage of running tests in batches.
+
+
+Catch2 also supports [splitting tests in a binary into multiple
+shards](command-line.md#test-sharding). This can be used by any test
+runner to run batches of tests in parallel. Do note that when selecting
+on the number of shards, you should have more shards than there are cores,
+to avoid issues with long running tests getting accidentally grouped in
+the same shard, and causing long-tailed execution time.
+
+**Note that naively composing sharding and random ordering of tests will break.**
+
+Invoking Catch2 test executable like this
+
+```text
+./tests --order rand --shard-index 0 --shard-count 3
+./tests --order rand --shard-index 1 --shard-count 3
+./tests --order rand --shard-index 2 --shard-count 3
+```
+
+does not guarantee covering all tests inside the executable, because
+each invocation will have its own random seed, thus it will have its own
+random order of tests and thus the partitioning of tests into shards will
+be different as well.
+
+To do this properly, you need the individual shards to share the random
+seed, e.g.
+```text
+./tests --order rand --shard-index 0 --shard-count 3 --rng-seed 0xBEEF
+./tests --order rand --shard-index 1 --shard-count 3 --rng-seed 0xBEEF
+./tests --order rand --shard-index 2 --shard-count 3 --rng-seed 0xBEEF
+```
+
+
+## Organizing tests into binaries
+
+Both overly large and overly small test binaries can cause issues. Overly
+large test binaries have to be recompiled and relinked often, and the
+link times are usually also long. Overly small test binaries in turn pay
+significant overhead from linking against Catch2 more often per compiled
+test case, and also make it hard/impossible to run tests in batches.
+
+Because there is no hard and fast rule for the right size of a test binary,
+I recommend having 1:1 correspondence between libraries in project and test
+binaries. (At least if it is possible, in some cases it is not.) Having
+a test binary for each library in project keeps related tests together,
+and makes tests easy to navigate by reflecting the project's organizational
+structure.
+
+
+---
+
+[Home](Readme.md#top)