From 2a19ae16b87547f152f262e05a55c7b2b969807b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martin=20Ho=C5=99e=C5=88ovsk=C3=BD?= Date: Wed, 30 Aug 2023 15:40:31 +0200 Subject: [PATCH] Rewrite commandline test spec docs Closes #2738 --- docs/command-line.md | 109 +++++++++++++++++++++++++++++++++---------- 1 file changed, 84 insertions(+), 25 deletions(-) diff --git a/docs/command-line.md b/docs/command-line.md index a15a2131..77d6a727 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -85,43 +85,102 @@ Click one of the following links to take you straight to that option - or scroll 
<test-spec> ...
-Test cases, wildcarded test cases, tags and tag expressions are all passed directly as arguments. Tags are distinguished by being enclosed in square brackets. +By providing a test spec, you filter which tests will be run. If you call +Catch2 without any test spec, then it will run all non-hidden test +cases. A test case is hidden if it has the `[!benchmark]` tag, any tag +with a dot at the start, e.g. `[.]` or `[.foo]`. -If no test specs are supplied then all test cases, except "hidden" tests, are run. -A test is hidden by giving it any tag starting with (or just) a period (```.```) - or, in the deprecated case, tagged ```[hide]``` or given name starting with `'./'`. To specify hidden tests from the command line ```[.]``` or ```[hide]``` can be used *regardless of how they were declared*. +There are three basic test specs that can then be combined into more +complex specs: -Specs must be enclosed in quotes if they contain spaces. If they do not contain spaces the quotes are optional. + * Full test name, e.g. `"Test 1"`. -Wildcards consist of the `*` character at the beginning and/or end of test case names and can substitute for any number of any characters (including none). +This allows only test cases whose name is "Test 1". -Test specs are case insensitive. + * Wildcarded test name, e.g. `"*Test"`, or `"Test*"`, or `"*Test*"`. -If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precedence, however. -Inclusions and exclusions are evaluated in left-to-right order. +This allows any test case whose name ends with, starts with, or contains +in the middle the string "Test". Note that the wildcard can only be at +the start or end. -Test case examples: + * Tag name, e.g. `[some-tag]`. +This allows any test case tagged with "[some-tag]". Remember that some +tags are special, e.g. those that start with "." or with "!". + + +You can also combine the basic test specs to create more complex test +specs. You can + + * Concatenate specs to apply all of them, e.g. `[some-tag][other-tag]`. + +This allows test cases that are tagged with **both** "[some-tag]" **and** +"[other-tag]". A test case with just "[some-tag]" will not pass the filter, +nor will test case with just "[other-tag]". + + * Comma-join specs to apply any of them, e.g. `[some-tag],[other-tag]`. + +This allows test cases that are tagged with **either** "[some-tag]" **or** +"[other-tag]". A test case with both will obviously also pass the filter. + +Note that commas take precendence over simple concatenation. This means +that `[a][b],[c]` accepts tests that are tagged with either both "[a]" and +"[b]", or tests that are tagged with just "[c]". + + * Negate the spec by prepending it with `~`, e.g. `~[some-tag]`. + +This rejects any test case that is tagged with "[some-tag]". Note that +rejection takes precedence over other filters. + +Note that negations always binds to the following _basic_ test spec. +This means that `~[foo][bar]` negates only the "[foo]" tag and not the +"[bar]" tag. + +Note that when Catch2 is deciding whether to include a test, first it +checks whether the test matches any negative filters. If it does, +the test is rejected. After that, the behaviour depends on whether there +are positive filters as well. If there are no positive filters, all +remaining non-hidden tests are included. If there are positive filters, +only tests that match the positive filters are included. + +You can also match test names with special characters by escaping them +with a backslash (`"\"`), e.g. a test named `"Do A, then B"` is matched +by "Do A\, then B" test spec. Backslash also escapes itself. + + +### Examples + +Given these TEST_CASEs, ``` -thisTestOnly Matches the test case called, 'thisTestOnly' -"this test only" Matches the test case called, 'this test only' -these* Matches all cases starting with 'these' -exclude:notThis Matches all tests except, 'notThis' -~notThis Matches all tests except, 'notThis' -~*private* Matches all tests except those that contain 'private' -a* ~ab* abc Matches all tests that start with 'a', except those that - start with 'ab', except 'abc', which is included -~[tag1] Matches all tests except those tagged with '[tag1]' --# [#somefile] Matches all tests from the file 'somefile.cpp' +TEST_CASE("Test 1") {} + +TEST_CASE("Test 2", "[.foo]") {} + +TEST_CASE("Test 3", "[.bar]") {} + +TEST_CASE("Test 4", "[.][foo][bar]") {} ``` -Names within square brackets are interpreted as tags. -A series of tags form an AND expression whereas a comma-separated sequence forms an OR expression. e.g.: +this is the result of these filters +``` +./tests # Selects only the first test, others are hidden +./tests "Test 1" # Selects only the first test, other do not match +./tests ~"Test 1" # Selects no tests. Test 1 is rejected, other tests are hidden +./tests "Test *" # Selects all tests. +./tests [bar] # Selects tests 3 and 4. Other tests are not tagged [bar] +./tests ~[foo] # Selects test 1, because it is the only non-hidden test without [foo] tag +./tests [foo][bar] # Selects test 4. +./tests [foo],[bar] # Selects tests 2, 3, 4. +./tests ~[foo][bar] # Selects test 3. 2 and 4 are rejected due to having [foo] tag +./tests ~"Test 2"[foo] # Selects test 4, because test 2 is explicitly rejected +./tests [foo][bar],"Test 1" # Selects tests 1 and 4. +./tests "Test 1*" # Selects test 1, wildcard can match zero characters +``` -
[one][two],[three]
-This matches all tests tagged `[one]` and `[two]`, as well as all tests tagged `[three]` +_Note: Using plain asterisk on a command line can cause issues with shell +expansion. Make sure that the asterisk is passed to Catch2 and is not +interpreted by the shell._ -Test names containing special characters, such as `,` or `[` can specify them on the command line using `\`. -`\` also escapes itself. ## Choosing a reporter to use