* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
Instead of redoing the whole line where path was found, only the
directory part of the path is removed, instead of removing all
of the line before the path starts.
This results in slight change in how junit and sonarqube approvals
come out, and significant change in how TeamCity reporter approvals
come out. This latter difference is the reason for the change,
as now the lines with `testFailed` and `testIgnored` messages
are not completely butchered.
This is needed so that we can use conjunction and other logical
type traits to workaround issue with older GCC versions (8 and
below), when they run into types that have ambiguous constructor
from `0`, see e.g. #2571.
However, using conjunction and friends in the SFINAE constraint
in the template parameter breaks for C++20 and up, due to the new
comparison operator rewriting rules. With C++20, when the compiler
see `a == b`, it also tries `b == a` and collects overload set
for both of these expressions.
In Catch2, this means that e.g. `REQUIRE( 1 == 2 )` would lead
the compiler to check overloads for both `ExprLhs<int> == int`
and `int == ExprLhs<int>`. Since the overload set and SFINAE
constraints assume that `ExprLhs<T>` is always on the left side,
when the compiler tries to resolve the template parameters, all
hell breaks loose and the compilation fails.
By moving the SFINAE constraints to the return type, the compiler
can discard the switched expression without having to resolve
the complex SFINAE constraints, and thus everything works the
way it is supposed to.
Fixes#2571
This is primarily done to support new `std::*_ordering` types,
but the refactoring also supports any other type with this
property.
The compilation overhead is surprisingly low. Testing it with
clang on a Linux machine, compiling our SelfTest project takes
only 2-3% longer with these changes than it takes otherwise.
Closes#2555
They also got slapped with the `[approvals]` tag in the process,
because we have too many approval tests and want less of them,
and these particular tests don't bring much value.
Related to #2090
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
MinGW doesn't support `__try` and friends at all, while Clang
only supports it partially, and the test would require some
changes to make it work there. Since this is only a test, we can
afford to keep it MSVC-only.
Closes#2447
This might become potentially useful in the future, when we want
to provide the ability to forward jump generators, to be able to
simply reproduce specific input to a test.
I am not yet sure how it would work, but the basic idea is that
it could be similar to the `-c` switch for selecting specific
`SECTION` in a test case.
This is kinda messy, because there is no good way to signal to
the compiler that some code uses direct comparison of floating
point numbers intentionally, so instead we have to use diagnostic
pragmas.
We also have to over-suppress the test files, because Clang (and
possibly GCC) still issue warnings from template instantiation
even if the instantion site is under warning suppression, so the
template definition has to be under warning suppression as well.
Closes#2406
The cleanup also found out that custom translation for std-derived
exceptions test wasn't running properly, and fixed that.
We cannot enable the warning globally, because the tests contain
some functions that are unused by design -- e.g. when checking
stringification priority of StringMaker vs range fallback and so
on.
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This ended up being a surprisingly large refactoring, motivated
by removing a `const_cast` from `Config`'s handling of reporter
streams, forced by previous commit.
