This means that code such as
```cpp
TEST_CASE() {
SECTION("first") { SUCCEED(); }
auto _ = GENERATE(1, 2);
SECTION("second") { SUCCEED(); }
}
```
will run and report 3 assertions, 1 from section "first" and 2
from section "second". This also applies for greater and potentially
more confusing nesting, but fundamentally it is up to the user to
avoid overly complex and confusing nestings, just as with `SECTION`s.
The old behaviour of `GENERATE` as first thing in a `TEST_CASE`,
`GENERATE` not followed by a `SECTION`, etc etc should be unchanged.
Closes#1938
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
Some compilers, e.g. the Green Hills C++ compiler, react badly to the
appearance of std::exception_ptr, std::current_exception,
std::rethrow_exception and std::uncaught_exception(s). To allow usage of
Catch2 with these compilers when exceptions are disabled, hide the usage
of std::exception_ptr etc. when compiling with
CATCH_CONFIG_DISABLE_EXCEPTIONS.
* Successive executions of the same `GENERATE` macro (e.g. because
of a for loop) no longer lead to multiple nested generators.
* The same line can now contain multiple `GENERATE` macros without
issues.
Fixes#1913
This brings our output inline with GTest's. We do not handle skipped
tests properly, but that should be currently less important than
having the attribute exist with proper value for non-skipped tests.
Thanks @joda-01.
Closes#1899
The issue is caused by deleted `std::__detail::begin` declared in `bits/iterator_concepts.h`. This would be found by ADL, and because it is deleted, compilation would fail. This change makes it so that we SFINAE on `begin(std::declval<T>())` and `end(std::declval<T>())` being well-formed.
D:\vcpkg\toolsrc\include\catch2\catch.hpp(11285): warning C6330: 'char' passed as _Param_(1) when 'unsigned char' is required in call to 'isspace'.
D:\vcpkg\toolsrc\include\catch2\catch.hpp(11288): warning C6330: 'char' passed as _Param_(1) when 'unsigned char' is required in call to 'isspace'.
ISO/IEC 9899:2011:
"7.4 Character handling <ctype.h>"/1
[...] In all cases the argument is an int, the value of which shall be
representable as an unsigned char or shall equal the value of the macro
EOF. If the argument has any other value, the behavior is undefined.
This means if isspace was passed a character like ñ it could corrupt
memory without the static_cast to treat it as a positive value after
integral promotion (and C libraries commonly use the int index supplied
as a key into a table which result in out of bounds access if the
resulting int is negative).
Ideally, clang-tidy would be smart that if one alias of a warning
is suppressed, then the other one is suppressed as well, but as of
right now, it isn't. This means that for now we have to suppress
both aliases of this warning. Opened upstream issue to fix this:
https://bugs.llvm.org/show_bug.cgi?id=45859
Obviously, ideally clang-tidy would also not warn that we are calling
a vararg function when it is an unevaluated magic builtin, but that
also is not happening right now and I opened an issue for it:
https://bugs.llvm.org/show_bug.cgi?id=45860Closes#1921
--- Fixes ---
* Vector matchers now support initializer list literals better
--- Improvements ---
* Added support for `^` (bitwise xor) to `CHECK` and `REQUIRE`
--- Improvements ---
* Running tests in random order (`--order rand`) has been reworked significantly (#1908)
* Given same seed, all platforms now produce the same order
* Given same seed, the relative order of tests does not change if you select only a subset of them
* Vector matchers support custom allocators (#1909)
* `|` and `&` (bitwise or and bitwise and) are now supported in `CHECK` and `REQUIRE`
* The resulting type must be convertible to `bool`
--- Fixes ---
* Fixed computation of benchmarking column widths in ConsoleReporter (#1885, #1886)
* Suppressed clang-tidy's `cppcoreguidelines-pro-type-vararg` in assertions (#1901)
* It was a false positive trigered by the new warning support workaround
* Fixed bug in test specification parser handling of OR'd patterns using escaping (#1905)
--- Miscellaneous ---
* Worked around IBM XL's codegen bug (#1907)
* It would emit code for _destructors_ of temporaries in an unevaluated context
* Improved detection of stdlib's support for `std::uncaught_exceptions` (#1911)
Catch assumes std::uncaught_exceptions is available whenever C++17 is
available, but for macOS versions older than 10.12 this is not the case.
Instead of checking the C++ version, use a macro to check whether the
feature is available.
Previously a random test ordering was obtained by applying std::shuffle
to the tests in declaration order. This has two problems:
- It depends on the declaration order, so the order in which the tests
will be run will be platform-specific.
- When trying to debug accidental inter-test dependencies, it is helpful
to be able to find a minimal subset of tests which exhibits the issue.
However, any change to the set of tests being run will completely
change the test ordering, making it difficult or impossible to reduce
the set of tests being run in any reasonably efficient manner.
Therefore, change the randomization approach to resolve both these
issues.
Generate a random value based on the user-provided RNG seed. Convert
every test case to an integer by hashing a combination of that value
with the test name. Sort the test cases by this integer.
The test names and RNG are platform-independent, so this should be
consistent across platforms. Also, removing one test does not change
the integer value associated with the remaining tests, so they remain in
the same order.
To hash, use the FNV-1a hash, except with the basis being our randomly
selected value rather than the fixed basis set in the algorithm. Cannot
use std::hash, because it is important that the result be
platform-independent.
It did not clear out all of its internal state when switching from
one pattern to another, so when it should've escaped `,`, it took
its position from its position in the original user-provided string,
rather than its position in the current pattern.
Fixes#1905
CATCH_INTERNAL_IGNORE_BUT_WARN() introduced with b7b346c triggers
clang-tidy warning 'cppcoreguidelines-pro-type-vararg' for every usage
of assertion macros like CHECK() and REQUIRE(). Silence it via NOLINT
in the '#if defined(__clang__)' block only, as clang-tidy honors those.
The old code caused warnings to fire under MSVC, and Clang <3.8.
I could not find a GCC version where it worked, but I assume that it
did at some point.
This new code causes all of MSVC, GCC, Clang, in current versions,
to emit signed/unsigned comparison warning in test like this:
```cpp
TEST_CASE() {
int32_t i = -1;
uint32_t j = 1;
REQUIRE(i != j);
}
```
Where previously only MSVC would emit the warning.
Fixes#1880
Given that in the 2 or so years that matchers are thing nobody complained,
it seems that people do not actually write this sort of code, and the
possibility will be removed in v3. However, to avoid correctness bugs,
we will have to support this weird code in v2.
C++11 math requires _GLIBCXX_USE_C99_MATH_TR1 to be true with gcc/clang.
Also fixes an issue with uClibc-ng where __UCLIBC__ is defined in features.h but
that is not included here and is thus no-op.
Add both `[.]` and `[!hide]` tags when registering a hidden test case, as per documentation.
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
- Overrides added
- usages of push_back() replaced with emplace_back()
- Loop variable made const-refernce
- NULL replaced with nullptr
- Names used in the declaration and definition unified
- size() replaced with empty
- Identical cases merged
b77cec05c0 fixed this problem for tagging tests, so that a test
case tagged with `[.foo]` would be parsed as tagged with `[.][foo]`.
This does the same for the test spec parsing.
Fixes#1798
Copying a `ReusableStringStream` would lead to "double free" of
the stream, and thus it could be used in multiple places at the
same time, breaking the output.
On systems where std::chrono::steady_clock::period is not std::nano, benchmark tests fail to compile due to trying to convert analysis.samples from a vector of duration<double, clock::period> to a vector of std::chrono::duration<double, std::nano>.
Previously, each warning suppression was self-contained, with its
own pair of `SUPPRESS_X_WARNING` and `UNSUPPRESS_X_WARNING` macros.
This had the obvious advantage of being self-containing, but it
also meant that if we needed to suppress more than one warning
in a single place, then we would manipulate the compiler's warning
state multiple times, even though logically we would only need one
layer.
The new way of suppressing warnings in macros is to push compiler's
warning state with `CATCH_INTERNAL_START_WARNINGS_SUPPRESSION` macro,
then disable whatever macros we need with the
`CATCH_INTERNAL_SUPPRESS_X_WARNINGS` macro, and then return to the
previous state using `CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION`.
The JUnit report is improved in that:
* The message shows the testing condition, not the result
* The actual message has similar output than the console one
Now it no longer tries to be this weird hybrid between an owning
and non-owning reference, and is only ever non-owning. This is also
reflected in its interface, for example `StringRef::isNullTerminated`
is now public, and `StringRef::c_str()` has the precondition that it
is true.
Overview of the changes:
* The `StringRef::m_data` member has been completely removed, as it
had no more uses.
* `StringRef::isSubstring()` has been made public and renamed to
`StringRef::isNullTerminated()`, so that the name reflects what the
method actually does.
* `StringRef::currentData()` has been renamed to `StringRef::data()`,
to be in line with common C++ containers and container-alikes.
* `StringRef::c_str()` will no longer silently make copies. It instead
has a precondition that `isNullTerminated()` is true.
* If the user needs a null-terminated string, they should use the
`std::string` conversion operator and call `c_str()` on the resulting
`std::string`.
* Some small optimizations in various places.
* Basic functionality is now `constexpr`.
This should now properly handle small numbers which would previously
output something like `[0.00000000000000019, 0.00000000000000019]`,
which does not allow user to read the numbers properly.
Closes#1760
this warning was introduced by rework to support NTTPs
since we have implementation macro for NTTPs and normal template test cases
warning is going to be suppressed
Fixes#1762
Only works for exceptions that publicly derive from `std::exception`
and the matching is done exactly, including case and whitespace.
Closes#1649Closes#1728
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Your branch is up-to-date with 'origin/master'.
#
# Changes to be committed:
# modified: ../docs/matchers.md
# modified: ../include/internal/catch_capture_matchers.h
# modified: ../projects/CMakeLists.txt
# modified: ../projects/SelfTest/Baselines/compact.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/console.std.approved.txt
# modified: ../projects/SelfTest/Baselines/console.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/junit.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/xml.sw.approved.txt
# modified: ../projects/SelfTest/UsageTests/Matchers.tests.cpp
#
# Untracked files:
# ./
# ../clang-full/
# ../clang-test/
# ../clang10-build/
# ../coverage-build/
# ../gcc-build/
# ../gcc-full/
# ../include/internal/catch_matchers_exception.cpp
# ../include/internal/catch_matchers_exception.hpp
# ../misc-build/
# ../msvc-sln/
# ../notes.txt
# ../test-install/
#
It checks Knuth's _close enough with tolerance_ relationship, that
is `|lhs - rhs| <= epsilon * max(|lhs|, |rhs|)`, rather then the
_very close with tolerance_ relationship that can be written down as
`|lhs - rhs| <= epsilon * min(|lhs|, |rhs|)`.
This is because it is the more common model around the internet, and
as such is likely to be less surprising to the users. In the future
we might want to provide the other model as well.
Closes#1746
This means that if you nest multiple random generators inside one
test case, they will not return the same sequence of numbers.
Idea taken from #1736 by Amit Herman.
Closes#1736Closes#1734
In the future, we will also want to introduce our own
`uniform_int_distribution` and `uniform_real_distribution` to get
repeatable test runs across different platforms.
