TAP format requires all results to be reported.
Removed extraneous preferences function (handled by parent)
Incorporated fix from 3d9e7db2e0
Simplified total printing
This means that code such as
```cpp
TEST_CASE() {
SECTION("first") { SUCCEED(); }
auto _ = GENERATE(1, 2);
SECTION("second") { SUCCEED(); }
}
```
will run and report 3 assertions, 1 from section "first" and 2
from section "second". This also applies for greater and potentially
more confusing nesting, but fundamentally it is up to the user to
avoid overly complex and confusing nestings, just as with `SECTION`s.
The old behaviour of `GENERATE` as first thing in a `TEST_CASE`,
`GENERATE` not followed by a `SECTION`, etc etc should be unchanged.
Closes#1938
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
Some compilers, e.g. the Green Hills C++ compiler, react badly to the
appearance of std::exception_ptr, std::current_exception,
std::rethrow_exception and std::uncaught_exception(s). To allow usage of
Catch2 with these compilers when exceptions are disabled, hide the usage
of std::exception_ptr etc. when compiling with
CATCH_CONFIG_DISABLE_EXCEPTIONS.
* Successive executions of the same `GENERATE` macro (e.g. because
of a for loop) no longer lead to multiple nested generators.
* The same line can now contain multiple `GENERATE` macros without
issues.
Fixes#1913
This brings our output inline with GTest's. We do not handle skipped
tests properly, but that should be currently less important than
having the attribute exist with proper value for non-skipped tests.
Thanks @joda-01.
Closes#1899
The issue is caused by deleted `std::__detail::begin` declared in `bits/iterator_concepts.h`. This would be found by ADL, and because it is deleted, compilation would fail. This change makes it so that we SFINAE on `begin(std::declval<T>())` and `end(std::declval<T>())` being well-formed.
D:\vcpkg\toolsrc\include\catch2\catch.hpp(11285): warning C6330: 'char' passed as _Param_(1) when 'unsigned char' is required in call to 'isspace'.
D:\vcpkg\toolsrc\include\catch2\catch.hpp(11288): warning C6330: 'char' passed as _Param_(1) when 'unsigned char' is required in call to 'isspace'.
ISO/IEC 9899:2011:
"7.4 Character handling <ctype.h>"/1
[...] In all cases the argument is an int, the value of which shall be
representable as an unsigned char or shall equal the value of the macro
EOF. If the argument has any other value, the behavior is undefined.
This means if isspace was passed a character like ñ it could corrupt
memory without the static_cast to treat it as a positive value after
integral promotion (and C libraries commonly use the int index supplied
as a key into a table which result in out of bounds access if the
resulting int is negative).
Ideally, clang-tidy would be smart that if one alias of a warning
is suppressed, then the other one is suppressed as well, but as of
right now, it isn't. This means that for now we have to suppress
both aliases of this warning. Opened upstream issue to fix this:
https://bugs.llvm.org/show_bug.cgi?id=45859
Obviously, ideally clang-tidy would also not warn that we are calling
a vararg function when it is an unevaluated magic builtin, but that
also is not happening right now and I opened an issue for it:
https://bugs.llvm.org/show_bug.cgi?id=45860Closes#1921
--- Fixes ---
* Vector matchers now support initializer list literals better
--- Improvements ---
* Added support for `^` (bitwise xor) to `CHECK` and `REQUIRE`
--- Improvements ---
* Running tests in random order (`--order rand`) has been reworked significantly (#1908)
* Given same seed, all platforms now produce the same order
* Given same seed, the relative order of tests does not change if you select only a subset of them
* Vector matchers support custom allocators (#1909)
* `|` and `&` (bitwise or and bitwise and) are now supported in `CHECK` and `REQUIRE`
* The resulting type must be convertible to `bool`
--- Fixes ---
* Fixed computation of benchmarking column widths in ConsoleReporter (#1885, #1886)
* Suppressed clang-tidy's `cppcoreguidelines-pro-type-vararg` in assertions (#1901)
* It was a false positive trigered by the new warning support workaround
* Fixed bug in test specification parser handling of OR'd patterns using escaping (#1905)
--- Miscellaneous ---
* Worked around IBM XL's codegen bug (#1907)
* It would emit code for _destructors_ of temporaries in an unevaluated context
* Improved detection of stdlib's support for `std::uncaught_exceptions` (#1911)
Catch assumes std::uncaught_exceptions is available whenever C++17 is
available, but for macOS versions older than 10.12 this is not the case.
Instead of checking the C++ version, use a macro to check whether the
feature is available.
Previously a random test ordering was obtained by applying std::shuffle
to the tests in declaration order. This has two problems:
- It depends on the declaration order, so the order in which the tests
will be run will be platform-specific.
- When trying to debug accidental inter-test dependencies, it is helpful
to be able to find a minimal subset of tests which exhibits the issue.
However, any change to the set of tests being run will completely
change the test ordering, making it difficult or impossible to reduce
the set of tests being run in any reasonably efficient manner.
Therefore, change the randomization approach to resolve both these
issues.
Generate a random value based on the user-provided RNG seed. Convert
every test case to an integer by hashing a combination of that value
with the test name. Sort the test cases by this integer.
The test names and RNG are platform-independent, so this should be
consistent across platforms. Also, removing one test does not change
the integer value associated with the remaining tests, so they remain in
the same order.
To hash, use the FNV-1a hash, except with the basis being our randomly
selected value rather than the fixed basis set in the algorithm. Cannot
use std::hash, because it is important that the result be
platform-independent.
It did not clear out all of its internal state when switching from
one pattern to another, so when it should've escaped `,`, it took
its position from its position in the original user-provided string,
rather than its position in the current pattern.
Fixes#1905
CATCH_INTERNAL_IGNORE_BUT_WARN() introduced with b7b346c triggers
clang-tidy warning 'cppcoreguidelines-pro-type-vararg' for every usage
of assertion macros like CHECK() and REQUIRE(). Silence it via NOLINT
in the '#if defined(__clang__)' block only, as clang-tidy honors those.
