catch2/docs/command-line.md

Command line

Contents
Specifying which tests to run
Choosing a reporter to use
Breaking into the debugger
Showing results for successful tests
Aborting after a certain number of failures
Listing available tests, tags or reporters
Sending output to a file
Naming a test run
Eliding assertions expected to throw
Make whitespace visible
Warnings
Reporting timings
Load test names to run from a file
Just test names
Specify the order test cases are run
Specify a seed for the Random Number Generator
Identify framework and version according to the libIdentify standard
Wait for key before continuing
Specify the number of benchmark samples to collect
Specify the number of resamples for bootstrapping
Specify the confidence-interval for bootstrapping
Disable statistical analysis of collected benchmark samples
Specify the amount of time in milliseconds spent on warming up each test
Usage
Specify the section to run
Filenames as tags
Override output colouring
Test Sharding
Allow running the binary without tests

Catch works quite nicely without any command line options at all - but for those times when you want greater control the following options are available. Click one of the following links to take you straight to that option - or scroll on to browse the available options.

<test-spec> ...
-h, -?, --help
-l, --list-tests
-t, --list-tags
-s, --success
-b, --break
-e, --nothrow
-i, --invisibles
-o, --out
-r, --reporter
-n, --name
-a, --abort
-x, --abortx
-w, --warn
-d, --durations
-f, --input-file
-c, --section
-#, --filenames-as-tags

--list-test-names-only
--list-reporters
--order
--rng-seed
--libidentify
--wait-for-keypress
--benchmark-samples
--benchmark-resamples
--benchmark-confidence-interval
--benchmark-no-analysis
--benchmark-warmup-time
--use-colour
--shard-count
--shard-index
--allow-running-no-tests

Specifying which tests to run

<test-spec> ...

Test cases, wildcarded test cases, tags and tag expressions are all passed directly as arguments. Tags are distinguished by being enclosed in square brackets.

If no test specs are supplied then all test cases, except "hidden" tests, are run. A test is hidden by giving it any tag starting with (or just) a period (.) - or, in the deprecated case, tagged [hide] or given name starting with './'. To specify hidden tests from the command line [.] or [hide] can be used regardless of how they were declared.

Specs must be enclosed in quotes if they contain spaces. If they do not contain spaces the quotes are optional.

Wildcards consist of the * character at the beginning and/or end of test case names and can substitute for any number of any characters (including none).

Test specs are case insensitive.

If a spec is prefixed with exclude: or the ~ character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precedence, however. Inclusions and exclusions are evaluated in left-to-right order.

Test case examples:

thisTestOnly            Matches the test case called, 'thisTestOnly'
"this test only"        Matches the test case called, 'this test only'
these*                  Matches all cases starting with 'these'
exclude:notThis         Matches all tests except, 'notThis'
~notThis                Matches all tests except, 'notThis'
~*private*              Matches all tests except those that contain 'private'
a* ~ab* abc             Matches all tests that start with 'a', except those that
                        start with 'ab', except 'abc', which is included
~[tag1]                 Matches all tests except those tagged with '[tag1]'
-# [#somefile]          Matches all tests from the file 'somefile.cpp'

Names within square brackets are interpreted as tags. A series of tags form an AND expression whereas a comma-separated sequence forms an OR expression. e.g.:

[one][two],[three]

This matches all tests tagged [one] and [two], as well as all tests tagged [three]

Test names containing special characters, such as , or [ can specify them on the command line using \. \ also escapes itself.

Choosing a reporter to use

-r, --reporter <reporter>

A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.
The bundled reporters are:

-r console
-r compact
-r xml
-r junit

The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Jenkins. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node.

Breaking into the debugger

-b, --break

Under most debuggers Catch2 is capable of automatically breaking on a test failure. This allows the user to see the current state of the test during failure.

Showing results for successful tests

-s, --success

Usually you only want to see reporting for failed tests. Sometimes it's useful to see all the output (especially when you don't trust that that test you just added worked first time!). To see successful, as well as failing, test results just pass this option. Note that each reporter may treat this option differently. The Junit reporter, for example, logs all results regardless.

Aborting after a certain number of failures

-a, --abort
-x, --abortx [<failure threshold>]

If a REQUIRE assertion fails the test case aborts, but subsequent test cases are still run. If a CHECK assertion fails even the current test case is not aborted.

Sometimes this results in a flood of failure messages and you'd rather just see the first few. Specifying -a or --abort on its own will abort the whole test run on the first failed assertion of any kind. Use -x or --abortx followed by a number to abort after that number of assertion failures.

Listing available tests, tags or reporters

-l, --list-tests
-t, --list-tags
--list-reporters

-l or --list-tests will list all registered tests, along with any tags. If one or more test-specs have been supplied too then only the matching tests will be listed.

-t or --list-tags lists all available tags, along with the number of test cases they match. Again, supplying test specs limits the tags that match.

--list-reporters lists the available reporters.

Sending output to a file

-o, --out <filename>

Use this option to send all output to a file. By default output is sent to stdout (note that uses of stdout and stderr from within test cases are redirected and included in the report - so even stderr will effectively end up on stdout).

Naming a test run

-n, --name <name for test run>

If a name is supplied it will be used by the reporter to provide an overall name for the test run. This can be useful if you are sending to a file, for example, and need to distinguish different test runs - either from different Catch executables or runs of the same executable with different options. If not supplied the name is defaulted to the name of the executable.

Eliding assertions expected to throw

-e, --nothrow

Skips all assertions that test that an exception is thrown, e.g. REQUIRE_THROWS.

These can be a nuisance in certain debugging environments that may break when exceptions are thrown (while this is usually optional for handled exceptions, it can be useful to have enabled if you are trying to track down something unexpected).

Sometimes exceptions are expected outside of one of the assertions that tests for them (perhaps thrown and caught within the code-under-test). The whole test case can be skipped when using -e by marking it with the [!throws] tag.

When running with this option any throw checking assertions are skipped so as not to contribute additional noise. Be careful if this affects the behaviour of subsequent tests.

Make whitespace visible

-i, --invisibles

If a string comparison fails due to differences in whitespace - especially leading or trailing whitespace - it can be hard to see what's going on. This option transforms tabs and newline characters into \t and \n respectively when printing.

Warnings

-w, --warn <warning name>

You can think of Catch2's warnings as the equivalent of -Werror (/WX) flag for C++ compilers. It turns some suspicious occurences, like a section without assertions, into errors. Because these might be intended, warnings are not enabled by default, but user can opt in.

You can enable multiple warnings at the same time.

There are currently two warnings implemented:

    NoAssertions        // Fail test case / leaf section if no assertions
                        // (e.g. `REQUIRE`) is encountered.
    UnmatchedTestSpec   // Fail test run if any of the CLI test specs did
                        // not match any tests.

UnmatchedTestSpec was introduced in Catch2 X.Y.Z.

Reporting timings

-d, --durations <yes/no>

When set to yes Catch will report the duration of each test case, in milliseconds. Note that it does this regardless of whether a test case passes or fails. Note, also, the certain reporters (e.g. Junit) always report test case durations regardless of this option being set or not.

-D, --min-duration <value>

--min-duration was introduced in Catch2 2.13.0

When set, Catch will report the duration of each test case that took more than <value> seconds, in milliseconds. This option is overriden by both -d yes and -d no, so that either all durations are reported, or none are.

Load test names to run from a file

-f, --input-file <filename>

Provide the name of a file that contains a list of test case names - one per line. Blank lines are skipped and anything after the comment character, #, is ignored.

A useful way to generate an initial instance of this file is to use the list-test-names-only option. This can then be manually curated to specify a specific subset of tests - or in a specific order.

Just test names

--list-test-names-only

This option lists all available tests in a non-indented form, one on each line. This makes it ideal for saving to a file and feeding back into the -f or --input-file option.

Specify the order test cases are run

--order <decl|lex|rand>

Test cases are ordered one of three ways:

decl

Declaration order (this is the default order if no --order argument is provided). Tests in the same TU are sorted using their declaration orders, different TUs are in an implementation (linking) dependent order.

lex

Lexicographic order. Tests are sorted by their name, their tags are ignored.

rand

Randomly sorted. The order is dependent on Catch2's random seed (see --rng-seed), and is subset invariant. What this means is that as long as the random seed is fixed, running only some tests (e.g. via tag) does not change their relative order.

The subset stability was introduced in Catch2 v2.12.0

Specify a seed for the Random Number Generator

--rng-seed <'time'|'random-device'|number>

Sets the seed for random number generators used by Catch2. These are used e.g. to shuffle tests when user asks for tests to be in random order.

Using time as the argument asks Catch2 generate the seed through call to std::time(nullptr). This provides very weak randomness and multiple runs of the binary can generate the same seed if they are started close to each other.

Using random-device asks for std::random_device to be used instead. If your implementation provides working std::random_device, it should be preferred to using time. Catch2 uses std::random_device by default.

Identify framework and version according to the libIdentify standard

--libidentify

See The LibIdentify repo for more information and examples.

Wait for key before continuing

--wait-for-keypress <never|start|exit|both>

Will cause the executable to print a message and wait until the return/ enter key is pressed before continuing - either before running any tests, after running all tests - or both, depending on the argument.

Specify the number of benchmark samples to collect

--benchmark-samples <# of samples>

Introduced in Catch2 2.9.0.

When running benchmarks a number of "samples" is collected. This is the base data for later statistical analysis. Per sample a clock resolution dependent number of iterations of the user code is run, which is independent of the number of samples. Defaults to 100.

Specify the number of resamples for bootstrapping

--benchmark-resamples <# of resamples>

Introduced in Catch2 2.9.0.

After the measurements are performed, statistical bootstrapping is performed on the samples. The number of resamples for that bootstrapping is configurable but defaults to 100000. Due to the bootstrapping it is possible to give estimates for the mean and standard deviation. The estimates come with a lower bound and an upper bound, and the confidence interval (which is configurable but defaults to 95%).

Specify the confidence-interval for bootstrapping

--benchmark-confidence-interval <confidence-interval>

Introduced in Catch2 2.9.0.

The confidence-interval is used for statistical bootstrapping on the samples to calculate the upper and lower bounds of mean and standard deviation. Must be between 0 and 1 and defaults to 0.95.

Disable statistical analysis of collected benchmark samples

--benchmark-no-analysis

Introduced in Catch2 2.9.0.

When this flag is specified no bootstrapping or any other statistical analysis is performed. Instead the user code is only measured and the plain mean from the samples is reported.

Specify the amount of time in milliseconds spent on warming up each test

--benchmark-warmup-time

Introduced in Catch2 2.11.2.

Configure the amount of time spent warming up each test.

Usage

-h, -?, --help

Prints the command line arguments to stdout

Specify the section to run

-c, --section <section name>

To limit execution to a specific section within a test case, use this option one or more times. To narrow to sub-sections use multiple instances, where each subsequent instance specifies a deeper nesting level.

E.g. if you have:

TEST_CASE( "Test" ) {
  SECTION( "sa" ) {
    SECTION( "sb" ) {
      /*...*/
    }
    SECTION( "sc" ) {
      /*...*/
    }
  }
  SECTION( "sd" ) {
    /*...*/
  }
}

Then you can run sb with:

./MyExe Test -c sa -c sb

Or run just sd with:

./MyExe Test -c sd

To run all of sa, including sb and sc use:

./MyExe Test -c sa

There are some limitations of this feature to be aware of:

• Code outside of sections being skipped will still be executed - e.g. any set-up code in the TEST_CASE before the start of the first section.
  
• At time of writing, wildcards are not supported in section names.
• If you specify a section without narrowing to a test case first then all test cases will be executed (but only matching sections within them).

Filenames as tags

-#, --filenames-as-tags

When this option is used then every test is given an additional tag which is formed of the unqualified filename it is found in, with any extension stripped, prefixed with the # character.

So, for example, tests within the file ~\Dev\MyProject\Ferrets.cpp would be tagged [#Ferrets].

Override output colouring

--use-colour <yes|no|auto>

Catch colours output for terminals, but omits colouring when it detects that output is being sent to a pipe. This is done to avoid interfering with automated processing of output.

--use-colour yes forces coloured output, --use-colour no disables coloured output. The default behaviour is --use-colour auto.

Test Sharding

--shard-count <#number of shards>, --shard-index <#shard index to run>

Introduced in Catch2 X.Y.Z.

When --shard-count <#number of shards> is used, the tests to execute will be split evenly in to the given number of sets, identified by indicies starting at 0. The tests in the set given by --shard-index <#shard index to run> will be executed. The default shard count is 1, and the default index to run is 0. It is an error to specify a shard index greater than the number of shards.

This is useful when you want to split test execution across multiple processes, as is done with Bazel test sharding.

Allow running the binary without tests

--allow-running-no-tests

Introduced in Catch2 X.Y.Z.

By default, Catch2 test binaries return non-0 exit code if no tests were run, e.g. if the binary was compiled with no tests, or the provided test spec matched no tests. This flag overrides that, so a test run with no tests still returns 0.

---

Home