* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
Instead of redoing the whole line where path was found, only the
directory part of the path is removed, instead of removing all
of the line before the path starts.
This results in slight change in how junit and sonarqube approvals
come out, and significant change in how TeamCity reporter approvals
come out. This latter difference is the reason for the change,
as now the lines with `testFailed` and `testIgnored` messages
are not completely butchered.
They also got slapped with the `[approvals]` tag in the process,
because we have too many approval tests and want less of them,
and these particular tests don't bring much value.
Related to #2090
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
This is kinda messy, because there is no good way to signal to
the compiler that some code uses direct comparison of floating
point numbers intentionally, so instead we have to use diagnostic
pragmas.
We also have to over-suppress the test files, because Clang (and
possibly GCC) still issue warnings from template instantiation
even if the instantion site is under warning suppression, so the
template definition has to be under warning suppression as well.
Closes#2406
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This ended up being a surprisingly large refactoring, motivated
by removing a `const_cast` from `Config`'s handling of reporter
streams, forced by previous commit.
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.
Closes#339 as it allows per-reporter forcing of ansi colour codes.
Test case hashing includes tags and class name
As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.
Closes#2304
This includes always compiling the ANSI and None colour
implementations, as they don't need to touch any platform
specific APIs, and removing their respective compile-time
configuration options.
Because the Win32 colour implementation requires Win32-specific
APIs, it is still hidden behind a compile-time toggle,
`CATCH_CONFIG_COLOUR_WIN32` (renamed from `..._COLOUR_WINDOWS`).
The commandline options for colours were also changed. The
option now uses different name, and allows to select between
different implementations, rather than changing whether
the compiled-in colour implementation is used through
"yes/no/default" options.
- do not hardcode content of containers
- prefix members with m_
- add const and non-const iterators to `with_mocked_iterator_access`
- remove `m_touched` as it wasn't filled properly and isn't used anyway
Removed:
* NaN normalization
* INFINITY normalization
* errno normalization
* Completely unused duration regex
Tests using these macros should be tagged `[approvals]`
so they are not run as part of approval tests.
Also simplified regex for the test's executable filename,
and hidden some tests relying on nullptr normalization.
This requires a bunch of different changes across the reporter
subsystem.
* We need to handle multiple reporters and their differing
preferences in `ListeningReporter`, e.g. what to do when
we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
By not materializing the lower cased tags ahead of time, we
save allocations at the cost of worsened performance when comparing
two tags.
Since there are rarely many tags, and commonly they are not
compared even if present, this is almost always a win. The new
implementation also improves the robustness of the code
responsible for handling tags in a case-insensitive manner.
Previously a lambda parser in Clara could only be invoked once,
even if it internally was ok with being invoked multiple times.
With this change, a lambda parser can mark itself as `accept_many`,
in which case it will be invoked multiple times if the appropriate
flag was supplied multiple times by the user.
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.
Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
