Commit Graph

306 Commits

Author SHA1 Message Date
Martin Hořeňovský
338e4ec1f8
Force disable console colours for tests that regex check output
Fixes #2458
2022-06-14 23:24:06 +02:00
Martin Hořeňovský
ff9506cedd
Only test SEH handling with MSVC
MinGW doesn't support `__try` and friends at all, while Clang
only supports it partially, and the test would require some
changes to make it work there. Since this is only a test, we can
afford to keep it MSVC-only.

Closes #2447
2022-06-06 00:29:01 +02:00
Sergio Losilla
5efd327dd4
Fix crash when running with --list-listeners and no registered listeners (#2442)
Co-authored-by: Sergio Losilla <sergio.losilla@nt-bnct.com>
2022-06-02 16:11:12 +02:00
Martin Hořeňovský
7a2a6c632f
Make Random*Generators be seedable
This is a necessary step to fix issue with combining multiple
random generators in one test case, with different section nesting.
2022-05-17 17:57:21 +02:00
Martin Hořeňovský
5d7883b551
Remove leftover comment in tests 2022-05-16 21:50:52 +02:00
Martin Hořeňovský
48f3226974
Add support for retrieving generator's element as string 2022-05-16 16:34:06 +02:00
Martin Hořeňovský
af8b54ecd5
Generators now keep track of the index of current element
This might become potentially useful in the future, when we want
to provide the ability to forward jump generators, to be able to
simply reproduce specific input to a test.

I am not yet sure how it would work, but the basic idea is that
it could be similar to the `-c` switch for selecting specific
`SECTION` in a test case.
2022-05-16 11:56:21 +02:00
Martin Hořeňovský
316025a0d8
Properly handle startup reporter registration errors 2022-05-15 10:35:12 +02:00
Martin Hořeňovský
33aeb603fe
Add check for registering multiple reporters under same name 2022-05-15 09:54:27 +02:00
Martin Hořeňovský
fc3d11b1d1
Add Wfloat-equal to default-enabled warnings
This is kinda messy, because there is no good way to signal to
the compiler that some code uses direct comparison of floating
point numbers intentionally, so instead we have to use diagnostic
pragmas.

We also have to over-suppress the test files, because Clang (and
possibly GCC) still issue warnings from template instantiation
even if the instantion site is under warning suppression, so the
template definition has to be under warning suppression as well.

Closes #2406
2022-05-14 15:47:20 +02:00
Martin Hořeňovský
1ef65d60f1
Tiny fixup in Misc.tests.cpp 2022-05-11 22:56:30 +02:00
Martin Hořeňovský
ed6b38b096
Specialize XML reporter's listener listing 2022-05-10 20:58:02 +02:00
Martin Hořeňovský
5a49285e9c
Add --list-listeners option 2022-05-10 20:57:59 +02:00
Martin Hořeňovský
d60fbe49be
Slight improvement for reporter test 2022-05-10 20:57:50 +02:00
Martin Hořeňovský
d9b0a38f81
Report Catch2's version in the xml reporter output 2022-05-03 19:46:35 +02:00
Martin Hořeňovský
91ea25e51a
Fix use of u8-literal that would break in C++20 2022-05-03 13:54:07 +02:00
Martin Hořeňovský
e2d07d35f4
Add -Wmissing-prototypes to common warnings 2022-05-03 13:54:05 +02:00
Martin Hořeňovský
d2cb934d28
Add -Wmissing-variable-declarations to common warnings 2022-05-03 13:54:03 +02:00
Martin Hořeňovský
198808a24e
Misc cleanups from exploring Clang warnings 2022-05-03 13:53:55 +02:00
Martin Hořeňovský
198713e5dc
Cleanup of unused functions
The cleanup also found out that custom translation for std-derived
exceptions test wasn't running properly, and fixed that.
We cannot enable the warning globally, because the tests contain
some functions that are unused by design -- e.g. when checking
stringification priority of StringMaker vs range fallback and so
on.
2022-05-02 21:32:24 +02:00
Martin Hořeňovský
c65e5b6514
Add 'uses-python' label to tests using Python 2022-04-24 13:31:28 +02:00
Martin Hořeňovský
07cdef2096
Fix uninit memory in Clara tests 2022-04-24 13:31:19 +02:00
Martin Hořeňovský
5baa29b6b9
Add test for %debug stream name 2022-04-23 23:34:27 +02:00
Daniel Feist
291b35b389
Added --skip-benchmarks command-line option. (#2408)
Closes #2392

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2022-04-23 23:14:49 +02:00
Martin Hořeňovský
17a04f88d9
Always report rng seed from builtin reporters
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.

CLoses #2065
2022-04-19 10:32:13 +02:00
Martin Hořeňovský
90e6905050
Remove no longer used CATCH_USE_VALGRIND cmake option 2022-04-18 12:14:21 +02:00
Martin Hořeňovský
7b93a2014c
Rename catch_stream -> catch_reusable_string_stream
After everything else was split out, this name much reflects the
actual contents of the file(s).
2022-04-16 16:34:08 +02:00
Martin Hořeňovský
98bb638fb2
Split out IStream out of catch_stream.hpp 2022-04-16 16:33:57 +02:00
Martin Hořeňovský
05e85c5652
Split out Catch::cout/cerr/clog into their own file 2022-04-16 16:33:50 +02:00
Martin Hořeňovský
c742ea9ad9
Some IWYU include cleanups 2022-04-14 17:16:01 +02:00
Martin Hořeňovský
a243cbae52
Introduce type to handle reporter specs with defaults filled-in 2022-04-13 15:02:25 +02:00
Martin Hořeňovský
79d1e82381
Store IStream instance owning ptrs in reporter instances
This ended up being a surprisingly large refactoring, motivated
by removing a `const_cast` from `Config`'s handling of reporter
streams, forced by previous commit.
2022-04-13 15:00:50 +02:00
Martin Hořeňovský
4f09f1120b
Make IStream::stream non-const
This way it makes much more sense from logically-const point
of view, and also means that concrete implementations don't
have to always have a `mutable` keyword on the stream member.
2022-04-13 14:59:39 +02:00
Martin Hořeňovský
be948f1fd0
Add COST property to the CMakeConfig tests 2022-04-10 21:55:52 +02:00
Martin Hořeňovský
4e006a93ff
Rename IStreamingReporterPtr => IEventListenerPtr
This reflects the renaming of the reporter/listener base class.
2022-04-10 21:50:51 +02:00
Lukasz Okraszewski
cb551b4f6d
Add default reporter for Bazel integration (#2399)
When the added Bazel configuration flag is enabled,
a default JUnit reporter will be added if the XML
envrioment variable is defined.
Fix include paths for generated config header.
Enable Bazel config by default when building with
Bazel.


Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2022-04-08 18:01:59 +02:00
Martin Hořeňovský
d5bfce4d84
Rename config::colourMode to config::defaultConfigMode
This describes the actual intention much better.
2022-04-07 12:50:13 +02:00
Martin Hořeňovský
423e1d2ebb
Integrate the new reporter specs into Catch2
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.

Closes #339 as it allows per-reporter forcing of ansi colour codes.
2022-04-07 12:50:08 +02:00
Martin Hořeňovský
3c06bcb374
Fix reporter parser spec to use the right key for colour-mode 2022-04-06 21:41:45 +02:00
Martin Hořeňovský
a51fd07bd0
Add helper for parsing the new reporter specs
The new reporter spec generalizes key-value options that can be
passed to the reporter, looking like this
`reporterName[::key=value]*`. A key can be either Catch2-recognized,
which currently means either `out` or `colour`, or reporter-specific
which is anything prefixed with `X`, e.g. `Xfoo`.
2022-04-05 15:45:54 +02:00
Martin Hořeňovský
8ac86495de
Cleanup ColourMode parsing in CLI 2022-04-05 13:42:21 +02:00
Martin Hořeňovský
d750da13a8
Split out colour mode string parsing into its own function 2022-04-05 13:42:06 +02:00
Martin Hořeňovský
c045733d05
Add splitReporterSpec helper 2022-04-05 12:57:33 +02:00
Martin Hořeňovský
797c3e7318
Refactor tests 2022-03-31 22:30:38 +02:00
Martin Hořeňovský
6206db5a73
Fix missing licence headers 2022-03-31 20:04:08 +02:00
Daniel Feist
78e33ce51f
Added TestCaseInfoHasher and tests. (#2394)
Test case hashing includes tags and class name

As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.

Closes #2304
2022-03-31 16:46:41 +02:00
Martin Hořeňovský
1a8a793178
Refactor colour handling to prepare for per-reporter colour modes
This includes always compiling the ANSI and None colour
implementations, as they don't need to touch any platform
specific APIs, and removing their respective compile-time
configuration options.

Because the Win32 colour implementation requires Win32-specific
APIs, it is still hidden behind a compile-time toggle,
`CATCH_CONFIG_COLOUR_WIN32` (renamed from `..._COLOUR_WINDOWS`).

The commandline options for colours were also changed. The
option now uses different name, and allows to select between
different implementations, rather than changing whether
the compiled-in colour implementation is used through
"yes/no/default" options.
2022-03-28 13:10:13 +02:00
Martin Hořeňovský
6227ca317e
Add tests for ColourGuard 2022-03-24 16:02:56 +01:00
Martin Hořeňovský
4d8acafecb
Rename isStdout => isConsole and add doccomment 2022-03-24 16:02:51 +01:00
Kochetkov, Yuriy
cf6dd937ab Add test for FatalConditionHandler and stack unwinding
Signed-off-by: Kochetkov, Yuriy <yuriyx.kochetkov@intel.com>
2022-03-24 15:49:02 +01:00
Martin Hořeňovský
4acc520f76
Event listeners no longer take reporter config in constructor
This also required splitting out Listener factory from
the reporter factory hierarchy. In return, the listener
factories only need to take in `IConfig const*`, which
opens up further refactorings down the road in the colour
selection and implementation.
2022-03-18 00:36:18 +01:00
schallerr
81f612c96c
Refactor MatchersRanges test utils (#2386)
- do not hardcode content of containers
- prefix members with m_
- add const and non-const iterators to `with_mocked_iterator_access`
- remove `m_touched` as it wasn't filled properly and isn't used anyway
2022-03-09 23:13:37 +01:00
Martin Hořeňovský
913f79a661
Each reporter keeps its own colour implementation
This opens path to per-reporter colour output customization,
and fixes multiple issues with the old colour implementation.

Under the old implementation, using Win32-backed colouring
would always change the colour used by the console, even if the
actual output was written elsewhere, such as a file passed by
the `--out` flag. This will no longer happen, as the reporter's
colour impl will check that the reporter's stream is pointed
to console before trying to change the colours.

POSIX/ANSI colour implementation suffered a similar-ish issue,
in that it only wrote the colour escape codes into the default
output stream, even if the reporter asking for colouring was
actually writing to a completely different output stream.
2022-03-08 12:51:13 +01:00
Martin Hořeňovský
06f74a0f8e
Add %stderr and %stdout as recognized magic stream names 2022-03-07 10:59:27 +01:00
Martin Hořeňovský
61d0f7a9af
Pass the whole IStream wrapper into reporter
This will become useful when reworking colour support, because
Win32 colour support requires checking whether the output is
stdout, which is done through the `IStream` wrapper.
2022-03-06 16:34:13 +01:00
John Beard
f9facc1881 Add variadic CAPTURE tests
This makes sure that CAPTURE works when called with variadic arguments,
and also works that way when disabled.

The underlying fix to #2316 is not applicable (CAPTURE is already
variadic when disabled).

This is a port of 5e94498ed0 to the
devel branch.
2022-03-05 16:06:54 +01:00
Martin Hořeňovský
ff5b311898
Remove superfluous warning suppression in tests 2022-02-22 20:13:17 +01:00
Martin Hořeňovský
715cd25081
Add IStream::isStdOut 2022-02-22 15:45:51 +01:00
GravisZro
1d4b42ad7b Minor whitespace fix 2022-02-21 20:04:43 +01:00
Martin Hořeňovský
72f0372664
Add tests for CMake configure toggles passing to Catch2 build 2022-02-20 19:32:30 +01:00
Martin Hořeňovský
fc5552d27b
Push down handling of default reporter to Config's constructor
This simplifies the handling of default reporter in console
parsing, at the cost of making `Config`'s constructor responsible
for more things.
2022-02-17 23:02:30 +01:00
Martin Hořeňovský
a9ed2c235d
Rename MiscFunctions.cmake -> CatchMiscFunctions.cmake 2022-02-07 00:05:11 +01:00
Martin Hořeňovský
7a2f9f4633
Extend checkLicense.py to also check tests 2022-01-29 13:10:54 +01:00
Martin Hořeňovský
79e4cd1af4
Cleanup summary comments in ExtraTests 2022-01-29 00:40:03 +01:00
Martin Hořeňovský
635db2785f
Use the new licence header in ExtraTests's cpp files 2022-01-29 00:07:44 +01:00
Martin Hořeňovský
51888d360a
Use the new licence header in TestScripts 2022-01-29 00:04:49 +01:00
Martin Hořeňovský
f83332d89b
Use the new licence header in SelfTest's cpp files 2022-01-29 00:03:43 +01:00
Martin Hořeňovský
e53a75b425
Add integration test for listener-reporter event ordering 2022-01-28 00:00:52 +01:00
Martin Hořeňovský
4ff8b27bb6
Cleanups output normalization in ApprovalTests
Removed:
 * NaN normalization
 * INFINITY normalization
 * errno normalization
 * Completely unused duration regex

Tests using these macros should be tagged `[approvals]`
so they are not run as part of approval tests.

Also simplified regex for the test's executable filename,
and hidden some tests relying on nullptr normalization.
2022-01-27 21:11:51 +01:00
Martin Hořeňovský
d861e73f86
Handle full unique test identifiers in testRandomOrder.py 2022-01-27 21:11:37 +01:00
Martin Hořeňovský
dc86d51af2
Update version string to mention Catch2 2022-01-27 20:15:05 +01:00
Martin Hořeňovský
b6ff2c3dda
Error out reporter registration for reporter names with '::' 2022-01-09 19:01:14 +01:00
Martin Hořeňovský
103cb16696
Rename listening reporter to multi reporter 2022-01-05 00:00:28 +01:00
Alan Jowett
98a6c69e1e Switch from AddVectoredExceptionHandler to SetUnhandledExceptionFilter
This avoids issues with Catch2's handler firing too early, on
structured exceptions that would be handled later. This issue
meant that the old attempts at structured exception handling
were incompatible with Windows's ASan, because it throws
continuable `C0000005` exception, which it then handles.

With the new handling, Catch2 is only notified if nothing else,
including the debugger, has handled the exception.

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

Closes #2332
Closes #2286
Closes #898
2022-01-03 22:50:32 +01:00
Martin Hořeňovský
88b28ab592
Extra compilation checks for compiling without exceptions 2022-01-01 23:21:37 +01:00
Martin Hořeňovský
ef3374ed81
Add more macros to the DisabledMacros test 2022-01-01 16:45:28 +01:00
Martin Hořeňovský
93882f7fab Support '-' as output path for stdout 2022-01-01 14:16:12 +01:00
Martin Hořeňovský
fae0be25b3 Improve error handling in parsing reporter specs 2022-01-01 14:16:11 +01:00
Martin Hořeňovský
899554bff2 Add test for handling of reporter's assertion preferences 2022-01-01 14:15:44 +01:00
Martin Hořeňovský
b4efa4751a Add test for handling of Listeners requesting stdout capture 2022-01-01 14:13:48 +01:00
Martin Hořeňovský
22547a3c5f Add test for stdout pass-through behaviour in MultiReporter 2022-01-01 14:11:49 +01:00
Martin Hořeňovský
8baf9c05a3 Add more unit tests
* Call order of listeners/reporters in multireporter
* Adding listeners/reporters properly updates reporter preferences
2022-01-01 14:10:52 +01:00
Martin Jeřábek
ccd67b293d Add support for multiple parallel reporters
This requires a bunch of different changes across the reporter
subsystem.

* We need to handle multiple reporters and their differing
  preferences in `ListeningReporter`, e.g. what to do when
  we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
  how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
2022-01-01 14:02:23 +01:00
Martin Hořeňovský
6b55f5d780
Don't send full stdout so far to testCasePartialEnded event 2022-01-01 12:21:41 +01:00
Martin Hořeňovský
c9c3b74805
Remove some obsolete tests 2021-12-31 15:12:53 +01:00
Martin Jeřábek
8711b63a0a
Add tests for StringRef's copy operations being shallow 2021-12-29 20:20:29 +01:00
Martin Hořeňovský
72a09de236
Add tests for Optional's op== and != 2021-12-29 16:58:22 +01:00
Martin Hořeňovský
f00b6e2019
Add more tests for various less common tag-related cases 2021-12-28 00:22:53 +01:00
Martin Hořeňovský
45577a1f4c
Refactor implementation of case-insensitivity in tags
By not materializing the lower cased tags ahead of time, we
save allocations at the cost of worsened performance when comparing
two tags.

Since there are rarely many tags, and commonly they are not
compared even if present, this is almost always a win. The new
implementation also improves the robustness of the code
responsible for handling tags in a case-insensitive manner.
2021-12-28 00:20:55 +01:00
Martin Hořeňovský
cbb6764fb1
Add CaseInsensitiveEqualTo comparison type 2021-12-26 18:54:47 +01:00
Martin Hořeňovský
3cc0c033e4
Allow enabling multiple warnings in one invocation 2021-12-18 20:50:05 +01:00
Martin Hořeňovský
840acedf62
Implement warning for unmatched test specs 2021-12-18 20:50:03 +01:00
Martin Hořeňovský
9f2dca5384
Exit with non-0 return code if no tests were run
A new flag, `--allow-running-no-tests` was added to override this
behaviour if exit code of 0 was desired.

This change also made `-w NoTests` obsolete, and so it has been
removed.
2021-12-18 20:50:00 +01:00
Martin Hořeňovský
08939cc8bb
Error out early if invalid test specs are provided 2021-12-18 20:49:47 +01:00
Martin Hořeňovský
8cb8f0b08b
Sweep out some Wsign-conversion warnings 2021-11-23 23:16:08 +01:00
Martin Hořeňovský
d012735c6e
Add test for runtime behaviour of STATIC_CHECK 2021-11-15 20:06:29 +01:00
Morwenn
f41d761674
Add STATIC_CHECK and STATIC_CHECK_FALSE (#2318) 2021-11-15 00:28:27 +01:00
Martin Hořeňovský
9b4e69333f
Small cleanup in tests 2021-11-14 11:27:43 +01:00
Martin Hořeňovský
9200b4078b
Move reporter_registrars.hpp to reporters/ 2021-11-10 23:52:59 +01:00
Martin Hořeňovský
6603f1d972
Use case in names of default reporters 2021-11-10 23:32:01 +01:00