From 9dc229693d7035b2a09057d578a69d97afc8e572 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martin=20Ho=C5=99e=C5=88ovsk=C3=BD?= Date: Mon, 25 Apr 2022 21:43:10 +0200 Subject: [PATCH] Document the stability guarantees of '--order rand' option --- docs/command-line.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/command-line.md b/docs/command-line.md index a0bb67fd..2e3db85b 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -334,13 +334,19 @@ Lexicographic order. Tests are sorted by their name, their tags are ignored. ### rand -Randomly sorted. The order is dependent on Catch2's random seed (see +Randomly ordered. The order is dependent on Catch2's random seed (see [`--rng-seed`](#rng-seed)), and is subset invariant. What this means is that as long as the random seed is fixed, running only some tests (e.g. via tag) does not change their relative order. > The subset stability was introduced in Catch2 v2.12.0 +Since the random order was made subset stable, we promise that given +the same random seed, the order of test cases will be the same across +different platforms, as long as the tests were compiled against identical +version of Catch2. We reserve the right to change the relative order +of tests cases between Catch2 versions, but it is unlikely to happen often. + ## Specify a seed for the Random Number Generator