This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.
Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
With these changes, all these benchmarks
```cpp
BENCHMARK("Empty benchmark") {};
BENCHMARK("Throwing benchmark") {
throw "just a plain literal, bleh";
};
BENCHMARK("Asserting benchmark") {
REQUIRE(1 == 2);
};
BENCHMARK("FAIL'd benchmark") {
FAIL("This benchmark only fails, nothing else");
};
```
report the respective failure and mark the outer `TEST_CASE` as
failed. Previously, the first two would not fail the `TEST_CASE`,
and the latter two would break xml reporter's formatting, because
`benchmarkFailed`, `benchmarkEnded` etc would not be be called
properly in failure cases.
This is a simplification of the fix proposed in #2152, with the
critical function split out so that it can be tested directly,
without having to go through the ULP matcher.
Closes#2152
A surrogate TU is TU that includes 1 specific header, and does
nothing else. This is useful to verify that the header is
self-sufficient and does not require other headers to be included
for compilation to succeed.
Closes#2106Closes#2166 (this is a better solution)
The problem was that under specific circumstances, namely that none
of their children progressed, `GeneratorTracker` will not progress.
This was changed recently, to allow for code like this, where a
`SECTION` follows a `GENERATE` at the same level:
```cpp
SECTION("A") {}
auto a = GENERATE(1, 2);
SECTION("B") {}
```
However, this interacted badly with `SECTION` filters (`-c foo`),
as they could deactivate all `SECTION`s below a generator, and thus
stop it from progressing forever. This commit makes GeneratorTracker
check whether there are any filters active, and if they are, it checks
whether its section-children can ever run.
Fixes#2025
The new output (mostly) follows the old `--list-test-names-only`
format, with the exception of no longer supporting line output
for `--verbosity high`.
Fixes#2051
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
This simplified variant supports only a subset of the functionality
in `std::unique_ptr<T>`. `Catch::Detail::unique_ptr<T>` only supports
single element pointer (no array support) with default deleter.
By removing the support for custom deleters, we also avoid requiring
significant machinery to support EBO, speeding up instantiations of
`unique_ptr<T>` significantly. Catch2 also currently does not need
to support `unique_ptr<T[]>`, so that is not supported either.
There are two reasons for this:
1) It is highly unlikely that someone has use for this header,
which has no customization points and only provides simplest
possible main, and cannot link the static library which also
provides a default main implementation.
2) It being a header was causing extra complications with
the convenience headers, and our checking script. This would either
require special handling in the checking script, or would break user's
of the main convenience header.
All in all, it is simpler and better in the long term to remove it,
than to fix its problems.
I do not think we need a safeguard against not including files in
CMake anymore, and as it is, it caused annoying false positive about
the default main implementation.
Thanks to the changes to compilation model, the tests for benchmarking
macros have been made part of the normal test run. This means that
the only purpose these separately compiled tests served was to waste
CI time.
When running tests in parallel, CTest runs the tests in decreasing
order of cost (time required), to get the largest speed up from
parallelism. However, the initial cost estimates for all tests are
0, and they are only updated after a test run. This works on a dev
machine, where the tests are ran over and over again, because
eventually the estimates become quite precise, but CI always does
a clean build with 0 estimates.
Because we have 2 slow tests, we want them to run first to avoid
losing parallelism. To do this, we provide them with a cost estimate
manually.
