This means that the user will see the estimation of full benchmark
running time when it is available, unlike now when it often only
ends up flushed after the benchmark is fully finished.
This means that the user will almost immediately see the start
of table like this
```
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
Fill vector generated 100 54 3.0834 ms
```
This presents significant improvement in user experience especially
for long running benchmarks.
* AssertionEnd does not reset the assertion info yet. That is done after populateReaction. And reset assertion info would also reset the result disposition to normal, so that any uncaught exception would be reported as failure
* Approving test output changes due to added unit tests
* Unit tests to throw std::runtime_error instead of std::exception
* Add a unit test to test incomplete assertion handler
---------
Co-authored-by: Ross <ross.tang@gfo-x.com>
* Add missing include for VxWorks build.
std::min is defined in algorithm provides std::min. It appears to be transitively included for most platforms. For VxWorks however this explicit include is required.
* Add option CATCH_CONFIG_PREFIX_MESSAGES to selectively prefix message macros only.
In contrast to CATCH_CONFIG_PREFIX_ALL, this will only prefix the following macros:
I.e. INFO, UNSCOPED_INFO, WARN and CATCH_CAPTURE
This is mainly useful for codebases that use INFO or WARN for their own logging macros.
Specifically we turned `mean`, `classify_outliers`, `jackknife`,
into concrete functions that take only `const_iterator` from vecs,
instead of generic iterators over anything.
I also changed `resample` to take `const_iterator` instead of
plain `iterator`, and similar for `standard_deviation`, and
`analyse_samples`.
* Split out BenchmarkInfo and BenchmarkStats to their own header
* Outline BenchmarkStats<> declaration to separate header
* Split out TestRunInfo into its own header
These changes let us remove the large `interfaces_reporter.hpp`
include from `benchmark.hpp`, and replace it with
`interfaces_capture.hpp` in `run_context.hpp`.
I also cleaned out `interfaces_repoter.hpp` from reporter headers
that depend on `reporter_common_base.hpp`. This will not change
anything in the actual inclusion set, but makes it logically
more consistent.
To keep the compilation firewall effect, the implementations
are hidden behind a PIMPL. In this case it is probably not
worth it, but we can inline it later if needed.
I do not know if checking the tracker name or the tracker's file
part of the location first would provide better results, but
in the common case, the line part of the location check should be
rather unique, because different `SECTION`s will have different
source lines where they are defined.
I also propagated this same check into `ITracker::findChild`,
because this significantly improves performance of section tracking
in Debug builds -> 10% in macro benchmark heavily focused on section
tracking. In Release build there is usually no difference, because
the inliner will inline `NameAndLoc::operator==` into `findChild`,
and then eliminate the redundant check. (If the inliner decides
against, then this still improves the performance on average).
`NameAndLocationRef` is pretty large type, so even in release build,
it is unlikely to be passed in registers. In addition to the fact
that some platforms currently do not allow passing even small types
in register (Windows ABI!!), it is better to pass it as a ref,
effectively passing around a pointer.
This enables its inlining even without LTO, which in turns enables
callers to determine that two StringRefs are unequals with simple
comparison of two numbers, without any function calls.
Catch2 suppresses unused variable and equivalent warnings in a couple
of places, but most importantly, in the declaration of autoRegistrar
in test registry. This warning gets triggered by NVHPC compiler. The
current patch adds three macros, namely:
CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
CATCH_INTERNAL_SUPPRESS_UNUSED_VARIABLE_WARNINGS
CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
for the NVHPC Compiler which in particular prevents that warning from
occurring. The compiler is detected completely separately from the
other compilers in this patch, because from what I found out, NVHPC
defines __GNUC__ as well for some reason. (I suspect because it
advertises itself as GNU compatible.)
We also add a condition to make sure that the `__GNUC__` path is not
taken by the NVHPC compiler.
Also split out helpers for testing matcher ranges (types whose
begin/end/empty/etc require ADL lookup, types whose iteration
uses iterator + sentinel pair, etc) into their own file.
The CompactReporter changes save 21 (430764 -> 430743) allocations
when running the SelfTest binary in default configuration. They
save about 500 allocations when running the binary with `-s`.
This eliminates 1945 (432709 -> 430764) allocations from running
`./tests/SelfTest -o /dev/null`. In general terms, this saves
an allocation every time an unvisited `SECTION` is passed, which
means that the saved allocations are quadratic in number of sibling
(same level) `SECTION`s in a test case.
Now we delay allocating owning `NameAndLocation` instances until
we construct a new tracker (because a tracker's lifetime can be
significantly different from the underlying tracked-thing's name).
This saves 4239 allocations (436948 -> 432709) when running
`./tests/SelfTest -o /dev/null`, at some cost to code clarity
due to introducing a new ref type, `NameAndLocationRef`.
When running `./tests/SelfTest -o /dev/null`, this saves 1272
allocations (437439 -> 437167). In general, this saves multiple
allocations per end of an entered `SECTION`, if the section name
was too long for SSO, because `RunContext::sectionEnded` can then
move the section's name further down the callstack.
When running `./tests/SelfTest -o /dev/null`, this saves 468
allocations (438907 -> 437439). In general, this saves 1 allocation
every time an entered `SECTION` ends and the section name was long
enough to move out of the SSO buffer.
* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This is needed so that we can use conjunction and other logical
type traits to workaround issue with older GCC versions (8 and
below), when they run into types that have ambiguous constructor
from `0`, see e.g. #2571.
However, using conjunction and friends in the SFINAE constraint
in the template parameter breaks for C++20 and up, due to the new
comparison operator rewriting rules. With C++20, when the compiler
see `a == b`, it also tries `b == a` and collects overload set
for both of these expressions.
In Catch2, this means that e.g. `REQUIRE( 1 == 2 )` would lead
the compiler to check overloads for both `ExprLhs<int> == int`
and `int == ExprLhs<int>`. Since the overload set and SFINAE
constraints assume that `ExprLhs<T>` is always on the left side,
when the compiler tries to resolve the template parameters, all
hell breaks loose and the compilation fails.
By moving the SFINAE constraints to the return type, the compiler
can discard the switched expression without having to resolve
the complex SFINAE constraints, and thus everything works the
way it is supposed to.
Fixes#2571
* Add new `CATCH_CONFIG` option for using `std::getenv`, because PS does not support env vars
* Add PS to platforms that have disabled posix signals.
* Small workaround for PS toolchain bug that prevents it from compiling `std::set` with lambda based comparator.
This is primarily done to support new `std::*_ordering` types,
but the refactoring also supports any other type with this
property.
The compilation overhead is surprisingly low. Testing it with
clang on a Linux machine, compiling our SelfTest project takes
only 2-3% longer with these changes than it takes otherwise.
Closes#2555
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
* meson: run through muon's fmt to fix formatting
* meson: switch arrays to files
Allows muon to alphabetically sort files. switch headers back to arrays
as split() can only be used on strings.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
The Meson[1] build system makes it easier incorporate third-party
libaries into a project if they also build using Meson.
Let's add a minimal Meson build that's compatible with the CMake build,
along with a GitHub workflow to check that it builds and that at least
the simplest SelfTest runs.
The handling of catch_user_config.hpp is inspired by BUILD.bazel and
doesn't attempt to support any configuratons options. Such features
could be added later.
Meson strongly discourages using wildcards to specify sources, so the
source and header lists are copied from CMakeLists.txt.
Add a new test workflow to test the Meson builds. I was unable to get
these tests to pass with Ubuntu 20.04, so they use Ubuntu 22.04.
I'm neither a CMake nor a Meson expert, but the results seem to work for
me.
[1] https://mesonbuild.com/
Co-authored-by: Mike Crowe <mcrowe@brightsign.biz>
* Suppress clang-tidy *-avoid-c-arrays for TEMPLATE_TEST_CASE
* Made globalRegistrar `const` to avoid `cppcoreguidelines-avoid-non-const-global-variables`
Fixes#2095
