This PR introduces a new `TEST_CASE` macro called `TEST_CASE_PERSISTENT_FIXTURE`. `TEST_CASE_PERSISTENT_FIXTURE` offers the same functionality as `TEST_CASE_METHOD` except for one difference. The object on which the test method is invoked is only created once for all invocations of the test case. The object is created just after the `testCaseStarting` event is broadcast and the object is destroyed just before the `testCaseEnding` event is broadcast.
The main motivation for this new functionality is to allow `TEST_CASE`s to do expensive setup and teardown once per `TEST_CASE`, without having to resort to abusing event listeners or static function variables with manual initialization.
Implements #1602
---------
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
`max_digits10` is the number of decimal digits that the type _can_
represent, in other words, the number of decimal digits needed to
disambiguate any two floating point numbers when serialized to
a string.
* AssertionEnd does not reset the assertion info yet. That is done after populateReaction. And reset assertion info would also reset the result disposition to normal, so that any uncaught exception would be reported as failure
* Approving test output changes due to added unit tests
* Unit tests to throw std::runtime_error instead of std::exception
* Add a unit test to test incomplete assertion handler
---------
Co-authored-by: Ross <ross.tang@gfo-x.com>
* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
Instead of redoing the whole line where path was found, only the
directory part of the path is removed, instead of removing all
of the line before the path starts.
This results in slight change in how junit and sonarqube approvals
come out, and significant change in how TeamCity reporter approvals
come out. This latter difference is the reason for the change,
as now the lines with `testFailed` and `testIgnored` messages
are not completely butchered.
They also got slapped with the `[approvals]` tag in the process,
because we have too many approval tests and want less of them,
and these particular tests don't bring much value.
Related to #2090
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.
Closes#339 as it allows per-reporter forcing of ansi colour codes.
Test case hashing includes tags and class name
As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.
Closes#2304
Removed:
* NaN normalization
* INFINITY normalization
* errno normalization
* Completely unused duration regex
Tests using these macros should be tagged `[approvals]`
so they are not run as part of approval tests.
Also simplified regex for the test's executable filename,
and hidden some tests relying on nullptr normalization.
This requires a bunch of different changes across the reporter
subsystem.
* We need to handle multiple reporters and their differing
preferences in `ListeningReporter`, e.g. what to do when
we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
Previously a lambda parser in Clara could only be invoked once,
even if it internally was ok with being invoked multiple times.
With this change, a lambda parser can mark itself as `accept_many`,
in which case it will be invoked multiple times if the appropriate
flag was supplied multiple times by the user.
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.
Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
