Commit Graph

148 Commits

Author SHA1 Message Date
Martin Hořeňovský
4acc520f76
Event listeners no longer take reporter config in constructor
This also required splitting out Listener factory from
the reporter factory hierarchy. In return, the listener
factories only need to take in `IConfig const*`, which
opens up further refactorings down the road in the colour
selection and implementation.
2022-03-18 00:36:18 +01:00
schallerr
81f612c96c
Refactor MatchersRanges test utils (#2386)
- do not hardcode content of containers
- prefix members with m_
- add const and non-const iterators to `with_mocked_iterator_access`
- remove `m_touched` as it wasn't filled properly and isn't used anyway
2022-03-09 23:13:37 +01:00
Martin Hořeňovský
913f79a661
Each reporter keeps its own colour implementation
This opens path to per-reporter colour output customization,
and fixes multiple issues with the old colour implementation.

Under the old implementation, using Win32-backed colouring
would always change the colour used by the console, even if the
actual output was written elsewhere, such as a file passed by
the `--out` flag. This will no longer happen, as the reporter's
colour impl will check that the reporter's stream is pointed
to console before trying to change the colours.

POSIX/ANSI colour implementation suffered a similar-ish issue,
in that it only wrote the colour escape codes into the default
output stream, even if the reporter asking for colouring was
actually writing to a completely different output stream.
2022-03-08 12:51:13 +01:00
Martin Hořeňovský
06f74a0f8e
Add %stderr and %stdout as recognized magic stream names 2022-03-07 10:59:27 +01:00
Martin Hořeňovský
61d0f7a9af
Pass the whole IStream wrapper into reporter
This will become useful when reworking colour support, because
Win32 colour support requires checking whether the output is
stdout, which is done through the `IStream` wrapper.
2022-03-06 16:34:13 +01:00
Martin Hořeňovský
ff5b311898
Remove superfluous warning suppression in tests 2022-02-22 20:13:17 +01:00
Martin Hořeňovský
715cd25081
Add IStream::isStdOut 2022-02-22 15:45:51 +01:00
GravisZro
1d4b42ad7b Minor whitespace fix 2022-02-21 20:04:43 +01:00
Martin Hořeňovský
fc5552d27b
Push down handling of default reporter to Config's constructor
This simplifies the handling of default reporter in console
parsing, at the cost of making `Config`'s constructor responsible
for more things.
2022-02-17 23:02:30 +01:00
Martin Hořeňovský
7a2f9f4633
Extend checkLicense.py to also check tests 2022-01-29 13:10:54 +01:00
Martin Hořeňovský
f83332d89b
Use the new licence header in SelfTest's cpp files 2022-01-29 00:03:43 +01:00
Martin Hořeňovský
4ff8b27bb6
Cleanups output normalization in ApprovalTests
Removed:
 * NaN normalization
 * INFINITY normalization
 * errno normalization
 * Completely unused duration regex

Tests using these macros should be tagged `[approvals]`
so they are not run as part of approval tests.

Also simplified regex for the test's executable filename,
and hidden some tests relying on nullptr normalization.
2022-01-27 21:11:51 +01:00
Martin Hořeňovský
b6ff2c3dda
Error out reporter registration for reporter names with '::' 2022-01-09 19:01:14 +01:00
Martin Hořeňovský
103cb16696
Rename listening reporter to multi reporter 2022-01-05 00:00:28 +01:00
Alan Jowett
98a6c69e1e Switch from AddVectoredExceptionHandler to SetUnhandledExceptionFilter
This avoids issues with Catch2's handler firing too early, on
structured exceptions that would be handled later. This issue
meant that the old attempts at structured exception handling
were incompatible with Windows's ASan, because it throws
continuable `C0000005` exception, which it then handles.

With the new handling, Catch2 is only notified if nothing else,
including the debugger, has handled the exception.

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

Closes #2332
Closes #2286
Closes #898
2022-01-03 22:50:32 +01:00
Martin Hořeňovský
fae0be25b3 Improve error handling in parsing reporter specs 2022-01-01 14:16:11 +01:00
Martin Hořeňovský
8baf9c05a3 Add more unit tests
* Call order of listeners/reporters in multireporter
* Adding listeners/reporters properly updates reporter preferences
2022-01-01 14:10:52 +01:00
Martin Jeřábek
ccd67b293d Add support for multiple parallel reporters
This requires a bunch of different changes across the reporter
subsystem.

* We need to handle multiple reporters and their differing
  preferences in `ListeningReporter`, e.g. what to do when
  we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
  how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
2022-01-01 14:02:23 +01:00
Martin Hořeňovský
c9c3b74805
Remove some obsolete tests 2021-12-31 15:12:53 +01:00
Martin Jeřábek
8711b63a0a
Add tests for StringRef's copy operations being shallow 2021-12-29 20:20:29 +01:00
Martin Hořeňovský
72a09de236
Add tests for Optional's op== and != 2021-12-29 16:58:22 +01:00
Martin Hořeňovský
f00b6e2019
Add more tests for various less common tag-related cases 2021-12-28 00:22:53 +01:00
Martin Hořeňovský
45577a1f4c
Refactor implementation of case-insensitivity in tags
By not materializing the lower cased tags ahead of time, we
save allocations at the cost of worsened performance when comparing
two tags.

Since there are rarely many tags, and commonly they are not
compared even if present, this is almost always a win. The new
implementation also improves the robustness of the code
responsible for handling tags in a case-insensitive manner.
2021-12-28 00:20:55 +01:00
Martin Hořeňovský
cbb6764fb1
Add CaseInsensitiveEqualTo comparison type 2021-12-26 18:54:47 +01:00
Martin Hořeňovský
3cc0c033e4
Allow enabling multiple warnings in one invocation 2021-12-18 20:50:05 +01:00
Martin Hořeňovský
9f2dca5384
Exit with non-0 return code if no tests were run
A new flag, `--allow-running-no-tests` was added to override this
behaviour if exit code of 0 was desired.

This change also made `-w NoTests` obsolete, and so it has been
removed.
2021-12-18 20:50:00 +01:00
Martin Hořeňovský
8cb8f0b08b
Sweep out some Wsign-conversion warnings 2021-11-23 23:16:08 +01:00
Morwenn
f41d761674
Add STATIC_CHECK and STATIC_CHECK_FALSE (#2318) 2021-11-15 00:28:27 +01:00
Martin Hořeňovský
9b4e69333f
Small cleanup in tests 2021-11-14 11:27:43 +01:00
Martin Hořeňovský
9200b4078b
Move reporter_registrars.hpp to reporters/ 2021-11-10 23:52:59 +01:00
Martin Hořeňovský
6603f1d972
Use case in names of default reporters 2021-11-10 23:32:01 +01:00
Martin Hořeňovský
931f41b4d6
Add some tests for TextFlow::Column 2021-11-01 22:51:17 +01:00
Martin Hořeňovský
3c5c86a4e4
Add test for filtering out multiple initial values in filter gen 2021-10-28 11:26:53 +02:00
Martin Hořeňovský
12d14a3c63
Add support for multiply calling lambda parsers in Clara
Previously a lambda parser in Clara could only be invoked once,
even if it internally was ok with being invoked multiple times.

With this change, a lambda parser can mark itself as `accept_many`,
in which case it will be invoked multiple times if the appropriate
flag was supplied multiple times by the user.
2021-10-27 20:15:28 +02:00
Martin Hořeňovský
5ac1ffe9ee Improve shardIndex/Count cli argument parsing 2021-10-27 17:24:30 +02:00
Ben Dunkin
3087e19cc7 Allow test sharding for e.g. Bazel test sharding feature
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.

Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
2021-10-27 17:24:30 +02:00
Martin Hořeňovský
905bf438ae
Fix bad indentation calculation in the console reporter
The problem came from the console reporter trying to provide a
fancy linebreaking (primarily for things like `SCENARIO` or the
BDD macros), so that new lines start with extra indentation if
the text being line broken starts as "{text}: ".

The console reporter did not properly take into account cases
where the ": " part would already be in a later line, in which
case it would ask for non-sensical level of indentation (larger
than single line length).

We fixed this by also enforcing that the special indentation case
only triggers if the ": " is found early enough in the line, so
that we also avoid degenerate cases like this:
```
blablabla: F
           a
           n
           c
           y
           .
           .
           .
```

Fixes #2309
2021-10-25 15:21:28 +02:00
Alecto Irene Perez
06cf2a4724
Apply PR #2297 to devel branch (#2300)
* Apply PR #2297 to devel branch

It turns out that Issue #2272 partially affected the devel branch. When
building tests with C++20, the compiler emits a warning that top-level
comma expressions in array subscripts are depricated. Warnings are
treated as errors, so this caused the build to fail.

This commit adds localized warning suppression
in accordance with this recommendation here:
https://github.com/catchorg/Catch2/pull/2297#discussion_r720848392

Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>

* Fixed unknown pragma warning on old versions of gcc & clang

This commit fixes an unkwown pragma warning on older versions of GCC
and Clang. These older versions don't have a warning for depricated use
of the comma subscript. Because warning suppression is localized, and
only refers to the comma subscript warning, it doesn't affect compiler
warnings in other parts of the code.

Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>

* More #warning backwards compatibility fixes

Signed-off-by: Alecto Irene Perez <perez.cs@pm.me>
2021-10-21 15:47:21 +02:00
Martin Hořeňovský
b406ad52a7
Mark !mayfail tests as skipped in the JUnit reporter
Should fix #2116
2021-10-10 22:21:39 +02:00
Martin Hořeňovský
1d9696d22d
Cleanup reporting of rng seed in existing reporters 2021-10-09 00:02:30 +02:00
Martin Hořeňovský
200a487cf2
Add generateRandomSeed utility to generate randomness seed 2021-10-08 21:35:41 +02:00
Martin Hořeňovský
13670f535f
Add more tests for XmlWrite::write* members 2021-10-02 14:39:52 +02:00
Martin Hořeňovský
426954032f
Rename Contains string matcher builder to ContainsSubstring
The problem with the old name was that it collided with the
range matcher `Contains`, and it was not really possible to
disambiguate them just with argument types.

Closes #2131
2021-09-27 21:28:33 +02:00
Martin Hořeňovský
f02c2678a1
Use StringRef for className in TestCaseInfo 2021-09-27 19:07:45 +02:00
Martin Hořeňovský
21b99d6f58
Add StringRef::compare for three way comparison 2021-09-27 18:41:17 +02:00
Martin Hořeňovský
d42e7a23a0
Change startsWith(char) to take StringRef as argument 2021-09-27 14:55:28 +02:00
Martin Hořeňovský
7bb00a42be
Remove stub of obsolete test 2021-09-27 14:45:39 +02:00
Martin Hořeňovský
fb4153e05e
Multiple tests can have same name as long as their tags differ
This change also changes it so that test case macros using a
class name can have same name **and** tags as long as the
used class name differs.

Closes #1915
Closes #1999
2021-09-25 21:37:03 +02:00
Martin Hořeňovský
3f8cae8025
Add consistency-checking event listener to SelfTest
This event listener performs basic consistency checks (akin to
matching braces) on events that are passed to the listeners
when the `SelfTest` test binary is run.

The current checks are about nesting events (e.g. `testCaseStarting`
cannot be received before `testRunStarting`, `sectionStarting`
can only be received when a test case is active, etc), and matching
up counts of starting/ended events.

The simplicity means that it could be confused by starting/ended
events matching up but being out of order, e.g.
```
* test case A starting
* test case B ended
* test case B starting
* test case A ended
```
would be accepted, even though it is wrong. However, doing full
order checking would be much more implementation work, for relatively
little benefit, so it is left out for now.
2021-09-15 23:38:43 +02:00
Martin Hořeňovský
e5938007f7
Completely remove the testGroup events
This was done because they were 1:1 mapping to testRun events, and
I could not think of a reasonable way to make them their own thing.
2021-09-07 20:00:22 +02:00