Commit Graph

568 Commits

Author SHA1 Message Date
Martin Hořeňovský
8ac86495de
Cleanup ColourMode parsing in CLI 2022-04-05 13:42:21 +02:00
Martin Hořeňovský
d750da13a8
Split out colour mode string parsing into its own function 2022-04-05 13:42:06 +02:00
Martin Hořeňovský
c045733d05
Add splitReporterSpec helper 2022-04-05 12:57:33 +02:00
Martin Hořeňovský
9fea3d253f
Add missing include to catch_optional.hpp 2022-04-05 12:55:30 +02:00
Martin Hořeňovský
6206db5a73
Fix missing licence headers 2022-03-31 20:04:08 +02:00
Daniel Feist
78e33ce51f
Added TestCaseInfoHasher and tests. (#2394)
Test case hashing includes tags and class name

As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.

Closes #2304
2022-03-31 16:46:41 +02:00
Martin Hořeňovský
1a8a793178
Refactor colour handling to prepare for per-reporter colour modes
This includes always compiling the ANSI and None colour
implementations, as they don't need to touch any platform
specific APIs, and removing their respective compile-time
configuration options.

Because the Win32 colour implementation requires Win32-specific
APIs, it is still hidden behind a compile-time toggle,
`CATCH_CONFIG_COLOUR_WIN32` (renamed from `..._COLOUR_WINDOWS`).

The commandline options for colours were also changed. The
option now uses different name, and allows to select between
different implementations, rather than changing whether
the compiled-in colour implementation is used through
"yes/no/default" options.
2022-03-28 13:10:13 +02:00
Martin Hořeňovský
a4e4e82474
Always indent test names at 2 spaces with default listing
This makes the output slightly more parsable in cases where
using the xml output from xml reporter is impractical, e.g. from
CMake scripts.
2022-03-24 16:03:46 +01:00
Martin Hořeňovský
081a1e9aba
ColourGuard is no longer constructed engaged
Forcing it to be engaged explicitly, either via `op<<`, or by
`ColourGuard::engage`, fixes an issue with multiple `ColourGuard`s
being constructed in a single expression. Because the construction
of the `ColourGuard` instances can happen in arbitrary order,
colours would be applied in arbitrary order too. However, a chain
of `op<<`s has strict call orders, fixing this issue.
2022-03-24 16:02:54 +01:00
Martin Hořeňovský
4d8acafecb
Rename isStdout => isConsole and add doccomment 2022-03-24 16:02:51 +01:00
Kochetkov, Yuriy
2ce64d1d8f Fix disengage failure logs for FatalConditionHandlerGuard
FatalConditionHandlerGuard is used within RunContext::invokeActiveTestCase().
The intent of this guard is to avoid binary crash without failed test being
reported.
Still in case FatalConditionHandlerGuard destructor being called during stack
unwinding AND finds unexpected top-level filter for SEH unhandled exception,
the binary may still crash. As result of such crash the original exception
details are being hidden.

As the Catch2 provides only `CATCH_CATCH_ANON` macro, with no access to
exception details by design, looks like the best way to handle issue is to:
 - state requirements explicitly by `noexcept` specifier
 - use `Catch::cerr()` to print out possible issue notification

Signed-off-by: Kochetkov, Yuriy <yuriyx.kochetkov@intel.com>
2022-03-24 15:49:02 +01:00
Kochetkov, Yuriy
7882f7359e Remove unnecessary casts
May break the logic in case of WinAPI changes with no warnings

Signed-off-by: Kochetkov, Yuriy <yuriyx.kochetkov@intel.com>
2022-03-24 15:49:02 +01:00
Martin Hořeňovský
0e176c318b
Refactored colour handling
* POSIX colour impl is now compiled for all platforms.
* Deciding whether a colour impl should be picked is now stream
  dependent, and thus incompatible implementations can be removed
  immediately, rather than checking when the colour is being used.
2022-03-18 14:04:07 +01:00
Martin Hořeňovský
c1c72c7e05
Better doccomment for IEventListener 2022-03-18 10:30:48 +01:00
Martin Hořeňovský
06092f727d
Remove special handling of default stream for reporters
This fixes an issue where reporter with default-output to stdout
would think that it was given a stream _not_ backed by console,
thus not using colour.
2022-03-18 00:41:47 +01:00
Martin Hořeňovský
4acc520f76
Event listeners no longer take reporter config in constructor
This also required splitting out Listener factory from
the reporter factory hierarchy. In return, the listener
factories only need to take in `IConfig const*`, which
opens up further refactorings down the road in the colour
selection and implementation.
2022-03-18 00:36:18 +01:00
Alan Jowett
18c58667d7
Generate Visual Studio project files with constant project guids
By default, CMake derives a Visual Studio project GUID from the
file path but the GUID can be overridden via a property
(see https://gitlab.kitware.com/cmake/cmake/-/commit/c85367f4).

Using a non-constant GUID can cause problems if other projects/repos
want to reference the catch2 vcxproj files, so we force a constant GUID here.

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

Resolves: #2388
2022-03-18 00:21:12 +01:00
Martin Hořeňovský
913f79a661
Each reporter keeps its own colour implementation
This opens path to per-reporter colour output customization,
and fixes multiple issues with the old colour implementation.

Under the old implementation, using Win32-backed colouring
would always change the colour used by the console, even if the
actual output was written elsewhere, such as a file passed by
the `--out` flag. This will no longer happen, as the reporter's
colour impl will check that the reporter's stream is pointed
to console before trying to change the colours.

POSIX/ANSI colour implementation suffered a similar-ish issue,
in that it only wrote the colour escape codes into the default
output stream, even if the reporter asking for colouring was
actually writing to a completely different output stream.
2022-03-08 12:51:13 +01:00
Martin Hořeňovský
06f74a0f8e
Add %stderr and %stdout as recognized magic stream names 2022-03-07 10:59:27 +01:00
Martin Hořeňovský
61d0f7a9af
Pass the whole IStream wrapper into reporter
This will become useful when reworking colour support, because
Win32 colour support requires checking whether the output is
stdout, which is done through the `IStream` wrapper.
2022-03-06 16:34:13 +01:00
Martin Hořeňovský
9b01c404f5
Fix cli help tooltip for --use-colour 2022-03-06 16:28:30 +01:00
Martin Hořeňovský
f206162b2d
Fix some messy formatting in catch_stats 2022-03-06 16:28:28 +01:00
Martin Hořeňovský
05d4ec62c8
Add common ReporterBase as parent of the helper bases
This is useful as a centralized place for handling common reporter
problems like handling output streams, and soon also colour impl
handling.
2022-03-06 16:28:22 +01:00
Martin Hořeňovský
4dd5e2eece
Rename base reporter iface IStreamingReporter -> IEventListener 2022-03-06 16:28:21 +01:00
Martin Hořeňovský
715cd25081
Add IStream::isStdOut 2022-02-22 15:45:51 +01:00
Martin Hořeňovský
4396a9119f
Add some missing configuration includes 2022-02-20 19:32:25 +01:00
Martin Hořeňovský
bda4b7df84
Fix fallback stringifier config 2022-02-20 19:32:22 +01:00
Martin Hořeňovský
33ffc3b6fc
Allow passing compile-time configuration options through CMake 2022-02-18 11:15:08 +01:00
Martin Hořeňovský
fc5552d27b
Push down handling of default reporter to Config's constructor
This simplifies the handling of default reporter in console
parsing, at the cost of making `Config`'s constructor responsible
for more things.
2022-02-17 23:02:30 +01:00
Martin Hořeňovský
a9ed2c235d
Rename MiscFunctions.cmake -> CatchMiscFunctions.cmake 2022-02-07 00:05:11 +01:00
Martin Hořeňovský
c912f62fc4
Remove Windows.h related defines 2022-02-05 16:21:06 +01:00
Certseeds
9afb6ce138 feature: double -> const auto when static_cast in right
</subject>

Branch: devel

<type>:
- [ ] Bug fix
- [ ] Bug fix (Test)
- [x] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] This change requires a documentation update

<body>

<footer>

Signed-off-by: Certseeds <51754303+Certseeds@users.noreply.github.com>
2022-02-04 00:09:01 +01:00
Martin Hořeňovský
efb54926ee
Don't cache stdout handle in Windows console colour impl
The cached handle would become invalid if some other code, say
a user-provided test code, redirects stdout through `freopen`
or `_dup2`, which would then cause AppVerifier to complain.

Fixes #2345
2022-01-29 23:12:55 +01:00
Martin Hořeňovský
dc86d51af2
Update version string to mention Catch2 2022-01-27 20:15:05 +01:00
Martin Hořeňovský
b6ff2c3dda
Error out reporter registration for reporter names with '::' 2022-01-09 19:01:14 +01:00
Martin Hořeňovský
875299cff0
Remove unused member from Totals
At one point it was inserted there as the simplest way to smuggle
around an extra return value for specific errors in executing
tests. Since then, the error has been changed to be handled
differently, and the member became unused.
2022-01-07 10:34:25 +01:00
Martin Hořeňovský
fff494c10a
Tear out trivial_abi on unique_ptr
Turns out people are bad at not combining code compiled with GCC
and Clang, and the improvement from `trivial_abi` on `unique_ptr`
is not worth the maintenance cost of having this be an opt-in
change.

Closes #2344
2022-01-05 00:04:25 +01:00
Martin Hořeňovský
103cb16696
Rename listening reporter to multi reporter 2022-01-05 00:00:28 +01:00
Martin Hořeňovský
f4af9f6926
Bump version to v3.0.0 preview 4 2022-01-03 23:16:39 +01:00
Alan Jowett
98a6c69e1e Switch from AddVectoredExceptionHandler to SetUnhandledExceptionFilter
This avoids issues with Catch2's handler firing too early, on
structured exceptions that would be handled later. This issue
meant that the old attempts at structured exception handling
were incompatible with Windows's ASan, because it throws
continuable `C0000005` exception, which it then handles.

With the new handling, Catch2 is only notified if nothing else,
including the debugger, has handled the exception.

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

Closes #2332
Closes #2286
Closes #898
2022-01-03 22:50:32 +01:00
Martin Hořeňovský
eeee4a49af
Remove short flags from --list-tests and --list-tags 2022-01-02 21:23:29 +01:00
Martin Hořeňovský
b5547f2ef6 Passthrough stdout/err in multireporter's partial test end
This should provide the same overall stdout/err, but the new
output should feel "faster" for test cases that are entered
and exited multiple times (e.g. due to generators).
2022-01-01 14:20:23 +01:00
Martin Hořeňovský
93882f7fab Support '-' as output path for stdout 2022-01-01 14:16:12 +01:00
Martin Hořeňovský
fae0be25b3 Improve error handling in parsing reporter specs 2022-01-01 14:16:11 +01:00
Martin Hořeňovský
8baf9c05a3 Add more unit tests
* Call order of listeners/reporters in multireporter
* Adding listeners/reporters properly updates reporter preferences
2022-01-01 14:10:52 +01:00
Martin Jeřábek
ccd67b293d Add support for multiple parallel reporters
This requires a bunch of different changes across the reporter
subsystem.

* We need to handle multiple reporters and their differing
  preferences in `ListeningReporter`, e.g. what to do when
  we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
  how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
2022-01-01 14:02:23 +01:00
Martin Hořeňovský
6b55f5d780
Don't send full stdout so far to testCasePartialEnded event 2022-01-01 12:21:41 +01:00
Martin Jeřábek
f0a89b7345
Add ops == and != for Optional 2021-12-29 16:58:16 +01:00
Martin Hořeňovský
45577a1f4c
Refactor implementation of case-insensitivity in tags
By not materializing the lower cased tags ahead of time, we
save allocations at the cost of worsened performance when comparing
two tags.

Since there are rarely many tags, and commonly they are not
compared even if present, this is almost always a win. The new
implementation also improves the robustness of the code
responsible for handling tags in a case-insensitive manner.
2021-12-28 00:20:55 +01:00
Martin Hořeňovský
cbb6764fb1
Add CaseInsensitiveEqualTo comparison type 2021-12-26 18:54:47 +01:00
Martin Hořeňovský
156e6fdfa9
CaseInsensitiveLess takes args as StringRefs 2021-12-26 00:52:52 +01:00
Martin Hořeňovský
187bf6db2b
Split out CaseInsensitiveCmp into its own file 2021-12-26 00:35:46 +01:00
Martin Hořeňovský
3cc0c033e4
Allow enabling multiple warnings in one invocation 2021-12-18 20:50:05 +01:00
Martin Hořeňovský
840acedf62
Implement warning for unmatched test specs 2021-12-18 20:50:03 +01:00
Martin Hořeňovský
9f2dca5384
Exit with non-0 return code if no tests were run
A new flag, `--allow-running-no-tests` was added to override this
behaviour if exit code of 0 was desired.

This change also made `-w NoTests` obsolete, and so it has been
removed.
2021-12-18 20:50:00 +01:00
Martin Hořeňovský
602e484f02
Rename reportInvalidArguments -> reportInvalidTestSpec
This is has been what it actually does for a long time, but it
has not been renamed earlier due to API compatibility.
2021-12-18 20:49:50 +01:00
Martin Hořeňovský
08939cc8bb
Error out early if invalid test specs are provided 2021-12-18 20:49:47 +01:00
Martin Hořeňovský
3bfe900bbc
Move shard logic check after help flag check 2021-12-18 20:49:44 +01:00
Martin Hořeňovský
dcf9479c85
Counts internally use uint64_t instead of size_t
This ensures that even for 32 bit platforms, the assertion count
should not plausibly overflow.
2021-12-06 20:42:51 +01:00
Martin Hořeňovský
c49faa62dd
Let users of amalgamated distribution override the default main
Closes #2324
2021-12-02 13:49:16 +01:00
Martin Hořeňovský
153965a655
Delete useless MatcherMethod base class
All of its functionality can be moved into the `MatcherBase` class,
simplifying the code a bit and removing a warning about class with
virtual member functions but no virtual destructor.

Closes #2182 as it is no longer relevant.
2021-11-26 00:38:13 +01:00
Martin Hořeňovský
b9baae6d93
Cleanup StringRef impl a bit 2021-11-23 23:16:46 +01:00
Martin Hořeňovský
c95072408f
Add catch_sharding.hpp to CML 2021-11-23 23:16:17 +01:00
Martin Hořeňovský
8cb8f0b08b
Sweep out some Wsign-conversion warnings 2021-11-23 23:16:08 +01:00
SketchyLizard
2db1cf3404 Fix template args to avoid conversion warnings
Fix avoids a warning about sign conversion when included from a file compiled with -Werror=sign-conversion.
2021-11-20 21:38:01 +01:00
Morwenn
f41d761674
Add STATIC_CHECK and STATIC_CHECK_FALSE (#2318) 2021-11-15 00:28:27 +01:00
Martin Hořeňovský
edc2f6e8a3
Further refactoring of StreamingReporterBase 2021-11-14 11:41:29 +01:00
Martin Hořeňovský
b2ac27423a
Cleanup visibility in StreamingReporterBase 2021-11-14 11:41:27 +01:00
Martin Hořeňovský
a754cb9062
Make TestRunInfo constexpr 2021-11-14 11:41:26 +01:00
Martin Hořeňovský
5f38cc39fa
JUnit/SonarQube reporters use base's stream member 2021-11-14 11:41:25 +01:00
Martin Hořeňovský
b892ab133c
Cleanup visibility in CumulativeReporterBase 2021-11-14 11:41:24 +01:00
Martin Hořeňovský
0c9fe16537
Opt out JUnit/SonarQube reporter from expanding passing assertions
Closes #1966
2021-11-14 11:27:45 +01:00
Martin Hořeňovský
d02ea5adee
Cumulative reporter base can be customized to not expand assertions 2021-11-14 11:27:44 +01:00
Martin Hořeňovský
4d9bfb2951
Avoid static analysis warning in Approx 2021-11-14 11:27:42 +01:00
Martin Hořeňovský
9200b4078b
Move reporter_registrars.hpp to reporters/ 2021-11-10 23:52:59 +01:00
Martin Hořeňovský
6603f1d972
Use case in names of default reporters 2021-11-10 23:32:01 +01:00
Martin Hořeňovský
62d8913d67
Cumulative reporter base records benchmark results 2021-11-09 11:52:50 +01:00
Martin Hořeňovský
8780425385
Make reporter lookup case insensitive, registration case preserving
Previously registration was case preserving, but lookup used
lowercased reporter name, so a reporter whose name contained
upper case character could not be requested by the user.
2021-11-09 11:50:03 +01:00
Martin Hořeňovský
7800fe9708
Lift toLower(char) to header 2021-11-09 11:44:54 +01:00
Martin Hořeňovský
141e384c60
Fix missing include in reporter_registrars.hpp 2021-11-08 11:32:27 +01:00
Martin Hořeňovský
f1239b2045 Add doxygen doccomments to IStreamingReporter 2021-11-05 14:24:22 +01:00
Martin Hořeňovský
912df7df35
Fix quadratic runtime when linebreaking strings without newlines
The problem was that every line would iterate from current line
start position to the end of the string, looking for a newline
to break on, leading to accidentally quadratic runtime. With this
change, the code only ever searches up to the current line's
length and not more.

Credit to @jorgenpt for the fix suggestion.

Closes #2315
2021-11-04 00:23:56 +01:00
Martin Hořeňovský
70c4ec78fb
Improve comments and names in TextFlow::Column 2021-11-01 19:14:37 +01:00
Martin Hořeňovský
455ae0c561 Typedef Column::iterator as Column::const_iterator not vice versa 2021-10-31 13:01:41 +01:00
Martin Hořeňovský
2520ad4b6e Return const_iterator from Column::begin/end const
This is what should normally happen, even if it does not change
anything given that `Column::const_iterator` is currently a typedef
for `Column::iterator`.
2021-10-31 12:59:00 +01:00
Martin Hořeňovský
e539e1cb52
Move strings in Clara's result type 2021-10-29 23:04:24 +02:00
Martin Hořeňovský
12d14a3c63
Add support for multiply calling lambda parsers in Clara
Previously a lambda parser in Clara could only be invoked once,
even if it internally was ok with being invoked multiple times.

With this change, a lambda parser can mark itself as `accept_many`,
in which case it will be invoked multiple times if the appropriate
flag was supplied multiple times by the user.
2021-10-27 20:15:28 +02:00
Martin Hořeňovský
f17725a186
Split void_type into its own header and rename it to void_t 2021-10-27 20:01:13 +02:00
Martin Hořeňovský
5ac1ffe9ee Improve shardIndex/Count cli argument parsing 2021-10-27 17:24:30 +02:00
Ben Dunkin
3087e19cc7 Allow test sharding for e.g. Bazel test sharding feature
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.

Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
2021-10-27 17:24:30 +02:00
Martin Hořeňovský
6456ee8b01
Return Clara parsing error message by const-ref 2021-10-27 15:10:52 +02:00
Martin Hořeňovský
905bf438ae
Fix bad indentation calculation in the console reporter
The problem came from the console reporter trying to provide a
fancy linebreaking (primarily for things like `SCENARIO` or the
BDD macros), so that new lines start with extra indentation if
the text being line broken starts as "{text}: ".

The console reporter did not properly take into account cases
where the ": " part would already be in a later line, in which
case it would ask for non-sensical level of indentation (larger
than single line length).

We fixed this by also enforcing that the special indentation case
only triggers if the ": " is found early enough in the line, so
that we also avoid degenerate cases like this:
```
blablabla: F
           a
           n
           c
           y
           .
           .
           .
```

Fixes #2309
2021-10-25 15:21:28 +02:00
Anders Schau Knatten
22750cde0e Disable false positive from clang-tidy
Clang-tidy is smart enough to understand that the conditional is never
updated in the loop body. It will let you get away with it if it can
prove that the conditional is always false, but that is not always
possible.

Here is an example where it's not able to prove it, and thus gives a
false positive. This is a minimal reproduction of an actual case I hit
in production, where `function` is picking the function based on some
`constexpr` logic related to which type argument is currently being
tested.

```
int f();

TEMPLATE_TEST_CASE("reproduction", "", int) {
    const auto function = []() {
        return f;
    }();
    const int error = function();
    REQUIRE(error == 0); // clang-tidy complains: bugprone-infinite-loop
}
```

I did not choose to add this test to the test suite, since we're not
running `clang-tidy` in CI afaik. To run it manually, simply add the
snippet above somewhere and run clang-tidy with
`--checks=bugprone-infinite-loop`. Or see an example at
https://godbolt.org/z/4v8b8WexP.

The reason we get the infinite loop warning in the first place is the
conditional at the end of this `do`-loop. Ideally, this conditional
would just be `while(false)`, but the actual content of the
`REQUIRE`-statement has been included here too in order to not loose
warnings from signed/unsigned comparisons. In short, if you do
`REQUIRE(i < j)`, where `i` is a negative signed integer and `j` is an
unsigned integer, you're supposed to get a warning from
`-Wsign-compare`. Due to the decomposition in Catch2, you lose this
warning, which is why the content of the `REQUIRE` statement has been
added to the conditional to force the compiler to evaluate the actual
comparison as well.

This was discussed on Discord today, and an alternative approach (which
I don't have time to implement) would be to in the decomposition replace
the comparison operators with `cmp_less` and friends. These are C++20
though, and would have to be implemented manually. I am also not sure
it's a good idea to "magically" change the semantics of `<` when it's
used inside a `REQUIRE` macro.

Another alternative approach would be to trigger this warning in a
different way, by including the content of the `REQUIRE` macro in a
different way which doesn't affect the for loop. But I don't have enough
of an overview here to know where would be a good place and how to test
that I didn't break anything.
2021-10-21 22:45:19 +02:00
Martin Hořeňovský
b406ad52a7
Mark !mayfail tests as skipped in the JUnit reporter
Should fix #2116
2021-10-10 22:21:39 +02:00
Martin Hořeňovský
de67278e14
JUnit reporter uses only 3 decimal places when reporting durations
We used to use whatever precision we ended up having from C++'s
stdlib. However, some relatively popular tools, like Jenkins,
use Maven SureFire XML schema to validate JUnit test reports, and
Maven SureFire schema requires the duration to have at most 3
decimal places.

For compatibility, the JUnit reporter will now respect this
limitation.

Closes #2221
2021-10-10 22:10:48 +02:00
Martin Hořeňovský
1d9696d22d
Cleanup reporting of rng seed in existing reporters 2021-10-09 00:02:30 +02:00
Martin Hořeňovský
ed1f343a41
Provide random-device option for --rng-seed and make it default 2021-10-08 21:35:45 +02:00
Martin Hořeňovský
200a487cf2
Add generateRandomSeed utility to generate randomness seed 2021-10-08 21:35:41 +02:00
Martin Hořeňovský
fce42b62ad
Generate random rng seed if user did not specify one
Closes #2161
2021-10-08 21:35:29 +02:00
Martin Hořeňovský
4e6d306742
Rename Catch::Option to Optional 2021-10-05 20:39:28 +02:00