Catch2 suppresses unused variable and equivalent warnings in a couple
of places, but most importantly, in the declaration of autoRegistrar
in test registry. This warning gets triggered by NVHPC compiler. The
current patch adds three macros, namely:
CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
CATCH_INTERNAL_SUPPRESS_UNUSED_VARIABLE_WARNINGS
CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
for the NVHPC Compiler which in particular prevents that warning from
occurring. The compiler is detected completely separately from the
other compilers in this patch, because from what I found out, NVHPC
defines __GNUC__ as well for some reason. (I suspect because it
advertises itself as GNU compatible.)
We also add a condition to make sure that the `__GNUC__` path is not
taken by the NVHPC compiler.
Also split out helpers for testing matcher ranges (types whose
begin/end/empty/etc require ADL lookup, types whose iteration
uses iterator + sentinel pair, etc) into their own file.
The CompactReporter changes save 21 (430764 -> 430743) allocations
when running the SelfTest binary in default configuration. They
save about 500 allocations when running the binary with `-s`.
This eliminates 1945 (432709 -> 430764) allocations from running
`./tests/SelfTest -o /dev/null`. In general terms, this saves
an allocation every time an unvisited `SECTION` is passed, which
means that the saved allocations are quadratic in number of sibling
(same level) `SECTION`s in a test case.
Now we delay allocating owning `NameAndLocation` instances until
we construct a new tracker (because a tracker's lifetime can be
significantly different from the underlying tracked-thing's name).
This saves 4239 allocations (436948 -> 432709) when running
`./tests/SelfTest -o /dev/null`, at some cost to code clarity
due to introducing a new ref type, `NameAndLocationRef`.
When running `./tests/SelfTest -o /dev/null`, this saves 1272
allocations (437439 -> 437167). In general, this saves multiple
allocations per end of an entered `SECTION`, if the section name
was too long for SSO, because `RunContext::sectionEnded` can then
move the section's name further down the callstack.
When running `./tests/SelfTest -o /dev/null`, this saves 468
allocations (438907 -> 437439). In general, this saves 1 allocation
every time an entered `SECTION` ends and the section name was long
enough to move out of the SSO buffer.
* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This is needed so that we can use conjunction and other logical
type traits to workaround issue with older GCC versions (8 and
below), when they run into types that have ambiguous constructor
from `0`, see e.g. #2571.
However, using conjunction and friends in the SFINAE constraint
in the template parameter breaks for C++20 and up, due to the new
comparison operator rewriting rules. With C++20, when the compiler
see `a == b`, it also tries `b == a` and collects overload set
for both of these expressions.
In Catch2, this means that e.g. `REQUIRE( 1 == 2 )` would lead
the compiler to check overloads for both `ExprLhs<int> == int`
and `int == ExprLhs<int>`. Since the overload set and SFINAE
constraints assume that `ExprLhs<T>` is always on the left side,
when the compiler tries to resolve the template parameters, all
hell breaks loose and the compilation fails.
By moving the SFINAE constraints to the return type, the compiler
can discard the switched expression without having to resolve
the complex SFINAE constraints, and thus everything works the
way it is supposed to.
Fixes#2571
* Add new `CATCH_CONFIG` option for using `std::getenv`, because PS does not support env vars
* Add PS to platforms that have disabled posix signals.
* Small workaround for PS toolchain bug that prevents it from compiling `std::set` with lambda based comparator.
This is primarily done to support new `std::*_ordering` types,
but the refactoring also supports any other type with this
property.
The compilation overhead is surprisingly low. Testing it with
clang on a Linux machine, compiling our SelfTest project takes
only 2-3% longer with these changes than it takes otherwise.
Closes#2555
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
* meson: run through muon's fmt to fix formatting
* meson: switch arrays to files
Allows muon to alphabetically sort files. switch headers back to arrays
as split() can only be used on strings.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
The Meson[1] build system makes it easier incorporate third-party
libaries into a project if they also build using Meson.
Let's add a minimal Meson build that's compatible with the CMake build,
along with a GitHub workflow to check that it builds and that at least
the simplest SelfTest runs.
The handling of catch_user_config.hpp is inspired by BUILD.bazel and
doesn't attempt to support any configuratons options. Such features
could be added later.
Meson strongly discourages using wildcards to specify sources, so the
source and header lists are copied from CMakeLists.txt.
Add a new test workflow to test the Meson builds. I was unable to get
these tests to pass with Ubuntu 20.04, so they use Ubuntu 22.04.
I'm neither a CMake nor a Meson expert, but the results seem to work for
me.
[1] https://mesonbuild.com/
Co-authored-by: Mike Crowe <mcrowe@brightsign.biz>
* Suppress clang-tidy *-avoid-c-arrays for TEMPLATE_TEST_CASE
* Made globalRegistrar `const` to avoid `cppcoreguidelines-avoid-non-const-global-variables`
Fixes#2095
when a test executable is killed by a signal (e.g. when executed by
ctest with timeout), the reporter files are not flushed. this can lead
to incomplete (or empty) report files.
to avoid this we enable automatic flushing via `std::unitbuf`
compare #663
`catch_sharding.hpp` was using `std::min` without including `<algorithm>`.
This worked on most platforms, but it wasn't transitively included in
the VxWorks toolchain.
The compile time improvements from using combined TUs mostly isn't
worth the annoyance they cause with various IDE shortcuts, like
when switching between header and its impl. file.
Splitting them apart also fixes the issue of empty subdirs being
installed due to `foo/internal` folders that only contained the
combined TUs and no headers.
Closes#2457Closes#2463
* GCC5 compat: work around inherited constructor issues
Don't use inherited constructors, forward manually instead. This
basically reverts 61f803126d.
I believe that GCC5 does not implement P0136, a C++17 change that made
inherited constructors actually usable and was backported as a DR all
the way to C++11.
* GCC5 compat: bypass std::pair construction issue
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
Reported as an issue on Discord. I thought that by GCC 9, the
C++ frontend was fixed enough to support `_Pragma`-based
suppression correctly, but apparently I was wrong.
This fixes multiple issues with random generators, with the most
important one being that multiple nested generators could return
values from the same sequence, due to internal implementation
details of `GENERATE`, and how they interact with test case
paths.
The cost of doing this is that given this simple `TEST_CASE`,
```cpp
TEST_CASE("foo") {
auto i = GENERATE(take(10, random(0, 100));
SECTION("A") {
auto j = GENERATE(take(10, random(0, 100));
}
SECTION("B") {
auto k = GENERATE(take(10, random(0, 100));
}
}
```
`k` will have different values between running the test as
a whole, e.g. with `./tests "foo"`, and running only the "B"
section with `./tests "foo" -c "B"`.
I consider this an acceptable cost, because the only alternative
would be very messy to implement, and add a lot of brittle and
complex code for relatively little benefit.
If this calculation changes, we will need to instead walk
the current tracker tree whenever a random generator is being
constructed, check for random generators on the path to root,
and take a seed from them.
This might become potentially useful in the future, when we want
to provide the ability to forward jump generators, to be able to
simply reproduce specific input to a test.
I am not yet sure how it would work, but the basic idea is that
it could be similar to the `-c` switch for selecting specific
`SECTION` in a test case.
The old instruction would cause the debugger to be stuck at the
triggering source line forever, while the new one should have the
expected semantics, where the debugger can then single-step,
continue. or generally do things, afterwards.
Closes#2422
This is kinda messy, because there is no good way to signal to
the compiler that some code uses direct comparison of floating
point numbers intentionally, so instead we have to use diagnostic
pragmas.
We also have to over-suppress the test files, because Clang (and
possibly GCC) still issue warnings from template instantiation
even if the instantion site is under warning suppression, so the
template definition has to be under warning suppression as well.
Closes#2406
The cleanup also found out that custom translation for std-derived
exceptions test wasn't running properly, and fixed that.
We cannot enable the warning globally, because the tests contain
some functions that are unused by design -- e.g. when checking
stringification priority of StringMaker vs range fallback and so
on.
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This ended up being a surprisingly large refactoring, motivated
by removing a `const_cast` from `Config`'s handling of reporter
streams, forced by previous commit.
This way it makes much more sense from logically-const point
of view, and also means that concrete implementations don't
have to always have a `mutable` keyword on the stream member.
When the added Bazel configuration flag is enabled,
a default JUnit reporter will be added if the XML
envrioment variable is defined.
Fix include paths for generated config header.
Enable Bazel config by default when building with
Bazel.
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
It has become completely vestigial, as it only ever passed-through
the argument down to a different function, and being private
member function, it didn't even introduce a useful compilation
firewall.
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.
Closes#339 as it allows per-reporter forcing of ansi colour codes.
The new reporter spec generalizes key-value options that can be
passed to the reporter, looking like this
`reporterName[::key=value]*`. A key can be either Catch2-recognized,
which currently means either `out` or `colour`, or reporter-specific
which is anything prefixed with `X`, e.g. `Xfoo`.
Test case hashing includes tags and class name
As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.
Closes#2304
This includes always compiling the ANSI and None colour
implementations, as they don't need to touch any platform
specific APIs, and removing their respective compile-time
configuration options.
Because the Win32 colour implementation requires Win32-specific
APIs, it is still hidden behind a compile-time toggle,
`CATCH_CONFIG_COLOUR_WIN32` (renamed from `..._COLOUR_WINDOWS`).
The commandline options for colours were also changed. The
option now uses different name, and allows to select between
different implementations, rather than changing whether
the compiled-in colour implementation is used through
"yes/no/default" options.
Forcing it to be engaged explicitly, either via `op<<`, or by
`ColourGuard::engage`, fixes an issue with multiple `ColourGuard`s
being constructed in a single expression. Because the construction
of the `ColourGuard` instances can happen in arbitrary order,
colours would be applied in arbitrary order too. However, a chain
of `op<<`s has strict call orders, fixing this issue.
FatalConditionHandlerGuard is used within RunContext::invokeActiveTestCase().
The intent of this guard is to avoid binary crash without failed test being
reported.
Still in case FatalConditionHandlerGuard destructor being called during stack
unwinding AND finds unexpected top-level filter for SEH unhandled exception,
the binary may still crash. As result of such crash the original exception
details are being hidden.
As the Catch2 provides only `CATCH_CATCH_ANON` macro, with no access to
exception details by design, looks like the best way to handle issue is to:
- state requirements explicitly by `noexcept` specifier
- use `Catch::cerr()` to print out possible issue notification
Signed-off-by: Kochetkov, Yuriy <yuriyx.kochetkov@intel.com>
* POSIX colour impl is now compiled for all platforms.
* Deciding whether a colour impl should be picked is now stream
dependent, and thus incompatible implementations can be removed
immediately, rather than checking when the colour is being used.
This fixes an issue where reporter with default-output to stdout
would think that it was given a stream _not_ backed by console,
thus not using colour.
This also required splitting out Listener factory from
the reporter factory hierarchy. In return, the listener
factories only need to take in `IConfig const*`, which
opens up further refactorings down the road in the colour
selection and implementation.
By default, CMake derives a Visual Studio project GUID from the
file path but the GUID can be overridden via a property
(see https://gitlab.kitware.com/cmake/cmake/-/commit/c85367f4).
Using a non-constant GUID can cause problems if other projects/repos
want to reference the catch2 vcxproj files, so we force a constant GUID here.
Signed-off-by: Alan Jowett <alanjo@microsoft.com>
Resolves: #2388
This opens path to per-reporter colour output customization,
and fixes multiple issues with the old colour implementation.
Under the old implementation, using Win32-backed colouring
would always change the colour used by the console, even if the
actual output was written elsewhere, such as a file passed by
the `--out` flag. This will no longer happen, as the reporter's
colour impl will check that the reporter's stream is pointed
to console before trying to change the colours.
POSIX/ANSI colour implementation suffered a similar-ish issue,
in that it only wrote the colour escape codes into the default
output stream, even if the reporter asking for colouring was
actually writing to a completely different output stream.
This will become useful when reworking colour support, because
Win32 colour support requires checking whether the output is
stdout, which is done through the `IStream` wrapper.
The cached handle would become invalid if some other code, say
a user-provided test code, redirects stdout through `freopen`
or `_dup2`, which would then cause AppVerifier to complain.
Fixes#2345
At one point it was inserted there as the simplest way to smuggle
around an extra return value for specific errors in executing
tests. Since then, the error has been changed to be handled
differently, and the member became unused.
