This requires a bunch of different changes across the reporter
subsystem.
* We need to handle multiple reporters and their differing
preferences in `ListeningReporter`, e.g. what to do when
we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
By not materializing the lower cased tags ahead of time, we
save allocations at the cost of worsened performance when comparing
two tags.
Since there are rarely many tags, and commonly they are not
compared even if present, this is almost always a win. The new
implementation also improves the robustness of the code
responsible for handling tags in a case-insensitive manner.
A new flag, `--allow-running-no-tests` was added to override this
behaviour if exit code of 0 was desired.
This change also made `-w NoTests` obsolete, and so it has been
removed.
Previously registration was case preserving, but lookup used
lowercased reporter name, so a reporter whose name contained
upper case character could not be requested by the user.
Previously a lambda parser in Clara could only be invoked once,
even if it internally was ok with being invoked multiple times.
With this change, a lambda parser can mark itself as `accept_many`,
in which case it will be invoked multiple times if the appropriate
flag was supplied multiple times by the user.
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.
Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
The problem came from the console reporter trying to provide a
fancy linebreaking (primarily for things like `SCENARIO` or the
BDD macros), so that new lines start with extra indentation if
the text being line broken starts as "{text}: ".
The console reporter did not properly take into account cases
where the ": " part would already be in a later line, in which
case it would ask for non-sensical level of indentation (larger
than single line length).
We fixed this by also enforcing that the special indentation case
only triggers if the ": " is found early enough in the line, so
that we also avoid degenerate cases like this:
```
blablabla: F
a
n
c
y
.
.
.
```
Fixes#2309
* Apply PR #2297 to devel branch
It turns out that Issue #2272 partially affected the devel branch. When
building tests with C++20, the compiler emits a warning that top-level
comma expressions in array subscripts are depricated. Warnings are
treated as errors, so this caused the build to fail.
This commit adds localized warning suppression
in accordance with this recommendation here:
https://github.com/catchorg/Catch2/pull/2297#discussion_r720848392
Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>
* Fixed unknown pragma warning on old versions of gcc & clang
This commit fixes an unkwown pragma warning on older versions of GCC
and Clang. These older versions don't have a warning for depricated use
of the comma subscript. Because warning suppression is localized, and
only refers to the comma subscript warning, it doesn't affect compiler
warnings in other parts of the code.
Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>
* More #warning backwards compatibility fixes
Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>
The problem with the old name was that it collided with the
range matcher `Contains`, and it was not really possible to
disambiguate them just with argument types.
Closes#2131
This change also changes it so that test case macros using a
class name can have same name **and** tags as long as the
used class name differs.
Closes#1915Closes#1999
This event listener performs basic consistency checks (akin to
matching braces) on events that are passed to the listeners
when the `SelfTest` test binary is run.
The current checks are about nesting events (e.g. `testCaseStarting`
cannot be received before `testRunStarting`, `sectionStarting`
can only be received when a test case is active, etc), and matching
up counts of starting/ended events.
The simplicity means that it could be confused by starting/ended
events matching up but being out of order, e.g.
```
* test case A starting
* test case B ended
* test case B starting
* test case A ended
```
would be accepted, even though it is wrong. However, doing full
order checking would be much more implementation work, for relatively
little benefit, so it is left out for now.
This means that e.g. for `TEST_CASE` with two sibling `SECTION`s
the event will fire twice, because the `TEST_CASE` will be entered
twice.
Closes#2107 (the event mentioned there already exists, but this
is its counterpart that we also want to provide to users)
With these changes, all these benchmarks
```cpp
BENCHMARK("Empty benchmark") {};
BENCHMARK("Throwing benchmark") {
throw "just a plain literal, bleh";
};
BENCHMARK("Asserting benchmark") {
REQUIRE(1 == 2);
};
BENCHMARK("FAIL'd benchmark") {
FAIL("This benchmark only fails, nothing else");
};
```
report the respective failure and mark the outer `TEST_CASE` as
failed. Previously, the first two would not fail the `TEST_CASE`,
and the latter two would break xml reporter's formatting, because
`benchmarkFailed`, `benchmarkEnded` etc would not be be called
properly in failure cases.
This is a simplification of the fix proposed in #2152, with the
critical function split out so that it can be tested directly,
without having to go through the ULP matcher.
Closes#2152
More specifically, made the actual implementation of string-like
type handling take argument as `Catch::StringRef`, instead of
taking `std::string const&`.
This means that string-like types that are not `std::string` no
longer need to pay for an extra construction of `std::string`
(including the potential allocation), before they can be stringified.
The actual string stringification routine is now also better about
reserving sufficient space.
This test tends to be brittle on Mac CI machines, which are
heavily loaded and bursty. Since the tests are only run as part
of the "extra tests" test set, this increase should not have
a significant impact on the total duration of CI runs.
A surrogate TU is TU that includes 1 specific header, and does
nothing else. This is useful to verify that the header is
self-sufficient and does not require other headers to be included
for compilation to succeed.
Closes#2106Closes#2166 (this is a better solution)
This includes
* Testing both positive and negative path through the matchers
* Testing them with types whose `begin` and `end` member functions
require ADL
* Testing them with types that return different types from `begin`
and `end`
The old name was a legacy of v2 era, where all headers were stitched
into one. With v3 using separate headers, it is better when they have
proper name.
The problem was that under specific circumstances, namely that none
of their children progressed, `GeneratorTracker` will not progress.
This was changed recently, to allow for code like this, where a
`SECTION` follows a `GENERATE` at the same level:
```cpp
SECTION("A") {}
auto a = GENERATE(1, 2);
SECTION("B") {}
```
However, this interacted badly with `SECTION` filters (`-c foo`),
as they could deactivate all `SECTION`s below a generator, and thus
stop it from progressing forever. This commit makes GeneratorTracker
check whether there are any filters active, and if they are, it checks
whether its section-children can ever run.
Fixes#2025
The new output (mostly) follows the old `--list-test-names-only`
format, with the exception of no longer supporting line output
for `--verbosity high`.
Fixes#2051
This commits also adds a script that does the amalgamation of headers
and .cpp files into the distributable version, removes the old
`generateSingleHeader` script, and also adds a very simple compilation
test for the amalgamated distribution.
Most of the changes are completely pointless renaming of constructor
arguments so that they do not use the same name as the type members,
but 🤷Closes#2015
The base was also renamed from `TestEventListenerBase` to
`EventListenerBase`, and modified to derive directly from the
reporter interface, rather than deriving from `StreamingReporterBase`.
Each of the two reporter bases now has its own header file, and
cpp file. Even though this adds another TU to the compilation,
the total CPU time taken by compilation is reduced by about 1%
for debug build and ~0.5% for optimized build of the main library.
(The improvement would be roughly doubles without splitting the TUs,
but the maintainability hit is not worth it.)
The code size of the static library build has also somewhat decreased.
Follow up: Introduce combined TU for reporters, and further split
apart the catch_reporter_streaming_base.hpp header into its
constituent parts, as it still contains a whole bunch of other stuff.
* Added some missing `noexcept`s on custom destructors.
* Fixed `std::move` being called on a const-reference.
* Initialized `ScopedMessage::m_moved` in class definition, instead
of doing so in constructors explicitly.
* Turned some `enum`s into `enum class`es.
* Initialized `StreamingReporterBase::currentTestCaseInfo` in class
definition.
* Some cleanups in SelfTest code.
TAP format requires all results to be reported.
Removed extraneous preferences function (handled by parent)
Incorporated fix from 3d9e7db2e0
Simplified total printing
This means that code such as
```cpp
TEST_CASE() {
SECTION("first") { SUCCEED(); }
auto _ = GENERATE(1, 2);
SECTION("second") { SUCCEED(); }
}
```
will run and report 3 assertions, 1 from section "first" and 2
from section "second". This also applies for greater and potentially
more confusing nesting, but fundamentally it is up to the user to
avoid overly complex and confusing nestings, just as with `SECTION`s.
The old behaviour of `GENERATE` as first thing in a `TEST_CASE`,
`GENERATE` not followed by a `SECTION`, etc etc should be unchanged.
Closes#1938
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
* Successive executions of the same `GENERATE` macro (e.g. because
of a for loop) no longer lead to multiple nested generators.
* The same line can now contain multiple `GENERATE` macros without
issues.
Fixes#1913
This brings our output inline with GTest's. We do not handle skipped
tests properly, but that should be currently less important than
having the attribute exist with proper value for non-skipped tests.
Thanks @joda-01.
Closes#1899
Doing some benchmarking with ClangBuildAnalyzer suggests that
compiling Catch2's `SelfTest` spends 10% of the time instantiating
`std::unique_ptr` for some interface types required for registering
and running tests.
The lesser compilation overhead of `Catch::Detail::unique_ptr` should
significantly reduce that time.
The compiled implementation was also changed to use the custom impl,
to avoid having to convert between using `std::unique_ptr` and
`Catch::Detail::unique_ptr`. This will likely also improve the compile
times of the implementation, but that is less important than improving
compilation times of the user's TUs with tests.
This simplified variant supports only a subset of the functionality
in `std::unique_ptr<T>`. `Catch::Detail::unique_ptr<T>` only supports
single element pointer (no array support) with default deleter.
By removing the support for custom deleters, we also avoid requiring
significant machinery to support EBO, speeding up instantiations of
`unique_ptr<T>` significantly. Catch2 also currently does not need
to support `unique_ptr<T[]>`, so that is not supported either.
As far as I know, only a few users actually use it, but these changes
allow us to avoid including a surprising amount of code in the main
compilation path.
There are two reasons for this:
1) It is highly unlikely that someone has use for this header,
which has no customization points and only provides simplest
possible main, and cannot link the static library which also
provides a default main implementation.
2) It being a header was causing extra complications with
the convenience headers, and our checking script. This would either
require special handling in the checking script, or would break user's
of the main convenience header.
All in all, it is simpler and better in the long term to remove it,
than to fix its problems.
I do not think we need a safeguard against not including files in
CMake anymore, and as it is, it caused annoying false positive about
the default main implementation.
Originally the tests were from #1912, but as it turned out, the issue
was somewhere else. Still, the inputs provided were interesting, so
they are now part of our test suite.
It did not clear out all of its internal state when switching from
one pattern to another, so when it should've escaped `,`, it took
its position from its position in the original user-provided string,
rather than its position in the current pattern.
Fixes#1905
The old code caused warnings to fire under MSVC, and Clang <3.8.
I could not find a GCC version where it worked, but I assume that it
did at some point.
This new code causes all of MSVC, GCC, Clang, in current versions,
to emit signed/unsigned comparison warning in test like this:
```cpp
TEST_CASE() {
int32_t i = -1;
uint32_t j = 1;
REQUIRE(i != j);
}
```
Where previously only MSVC would emit the warning.
Fixes#1880
These files are not included by the default
`#include <catch2/catch_test_macros.hpp>` path, so that users do
not have to pay for them if they do not use them. Follow up is to
split out the small part of `catch_preprocessor.hpp` used by the
default test macros (AFAIK, it is just `INTERNAL_CATCH_REMOVE_PARENS`
macro), so that it is not included by the default path either.
Also fixes#1892 by providing the missing macros.
This is both a really big and a really small commit. It is small in
that it only contains renaming, moving and modification of include
directives caused by this.
It is really big in the obvious way of touching something like 200
files.
The new rules for naming files is simple: headers use the `.hpp`
extension. The rules for physical file layout is still kinda in
progress, but the basics are also simple:
* Significant parts of functionality get their own subfolder
* Benchmarking is in `catch2/benchmark`
* Matchers are in `catch2/matchers`
* Generators are in `catch2/generators`
* Reporters are in `catch2/reporters`
* Baseline testing facilities are in `catch2/`
* Various top level folders also contain `internal` subfolder,
with files that users probably do not want to include directly,
at least not until they have to write something like their own
reporter.
* The exact files in these subfolders is likely to change later
on
Note that while some includes were cleaned up in this commit, it
is only the low hanging fruit and further cleanup using automatic
tooling will happen later.
Also note that various include guards, copyright notices and file
headers will also be standardized later, rather than in this commit.
This was an old "include all" header, that we no longer want to be
usable, to make the include differences in new versions explicit.
We will introduce new "include all" headers later, in the form of
`catch_all.hpp`, `catch_matchers_all.hpp` and so on...
The two changes are
`catch_matchers_templates` -> `catch_matchers_templated` and
`catch_matchers_generic` -> `catch_matchers_predicate`. The former
is mostly cosmetic, but the second was previously significantly
misleading, and as the library is now to be consumed by including
specific headers, this needed to be fixed.
`SizeIs` can accept both `size_t` and a matcher. In the first case,
it checks whether the size of the range is equal to specified size.
In the second case, it checks whether the provided matcher accepts
the size of the range.
In the future we can expect many more matchers, so let's give them
a place to live.
Also moved matcher-related internal files to `internal` subfolder.
Ideally we should sort out all of our source code, but that will
have to come later.
This commit also forbids composing lvalues of composed matchers, as
per previous deprecation notice. I do not expect this to be contentious
in practice, because there was a bug in that usage for years, and
nobody complained.
Thanks to the changes to compilation model, the tests for benchmarking
macros have been made part of the normal test run. This means that
the only purpose these separately compiled tests served was to waste
CI time.
Given that in the 2 or so years that matchers are thing nobody complained,
it seems that people do not actually write this sort of code, and the
possibility will be removed in v3. However, to avoid correctness bugs,
we will have to support this weird code in v2.
- Overrides added
- usages of push_back() replaced with emplace_back()
- Loop variable made const-refernce
- NULL replaced with nullptr
- Names used in the declaration and definition unified
- size() replaced with empty
- Identical cases merged
This commit extends the Matchers feature with the ability to have type-independent (e.g. templated) matchers. This is done by adding a new base type that Matchers can extend, `MatcherGenericBase`, and overloads of operators `!`, `&&` and `||` that handle matchers extending `MatcherGenericBase` in a special manner.
These new matchers can also take their arguments as values and non-const references.
Closes#1307Closes#1553Closes#1554
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
b77cec05c0 fixed this problem for tagging tests, so that a test
case tagged with `[.foo]` would be parsed as tagged with `[.][foo]`.
This does the same for the test spec parsing.
Fixes#1798
Now that Catch2 is a proper library, we can always build the full
library (comparatively minor slowdown) and the user can avoid
including benchmarking headers to avoid the compilation slowdown.
When running tests in parallel, CTest runs the tests in decreasing
order of cost (time required), to get the largest speed up from
parallelism. However, the initial cost estimates for all tests are
0, and they are only updated after a test run. This works on a dev
machine, where the tests are ran over and over again, because
eventually the estimates become quite precise, but CI always does
a clean build with 0 estimates.
Because we have 2 slow tests, we want them to run first to avoid
losing parallelism. To do this, we provide them with a cost estimate
manually.
Now that the recommended distribution and usage method is proper
library, users can just avoid including the matcher headers to get
basically the same effect.
