Rewrite event listener docs and add event documentation

This commit is contained in:
Martin Hořeňovský 2021-11-18 23:00:45 +01:00
parent dd12ce8141
commit 0ccb1c30c6
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
2 changed files with 200 additions and 56 deletions

View File

@ -1,74 +1,43 @@
<a id="top"></a> <a id="top"></a>
# Event Listeners # Event Listeners
A `Listener` is a class you can register with Catch that will then be passed events, An event listener is a bit like a reporter, in that it responds to various
such as a test case starting or ending, as they happen during a test run. reporter events in Catch2, but it is not expected to write any output.
`Listeners` are actually types of `Reporters`, with a few small differences: Instead, an event listener performs actions within the test process, such
as performing global initialization (e.g. of a C library), or cleaning out
1. Once registered in code they are automatically used - you don't need to specify them on the command line in-memory logs if they are not needed (the test case passed).
2. They are called in addition to (just before) any reporters, and you can register multiple listeners.
3. They derive from `Catch::TestEventListenerBase`, which has default stubs for all the events,
so you are not forced to implement events you're not interested in.
4. You register a listener with `CATCH_REGISTER_LISTENER`
Unlike reporters, each registered event listener is always active. Event
listeners are always notified before reporter(s).
## Implementing a Listener To write your own event listener, you should derive from `Catch::TestEventListenerBase`,
Simply derive a class from `Catch::TestEventListenerBase` and implement the methods you are interested in, either in as it provides empty stubs for all reporter events, allowing you to
the main source file (i.e. the one that defines `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`), or in a only override events you care for. Afterwards you have to register it
file that defines `CATCH_CONFIG_EXTERNAL_INTERFACES`. with Catch2 using `CATCH_REGISTER_LISTENER` macro, so that Catch2 knows
about it and instantiates it before running tests.
Then register it using `CATCH_REGISTER_LISTENER`. Example event listener:
```cpp
#include <catch2/reporters/catch_reporter_event_listener.hpp>
#include <catch2/catch_reporter_registrars.hpp>
For example ([complete source code](../examples/210-Evt-EventListeners.cpp)): class testRunListener : public Catch::EventListenerBase {
public:
using Catch::EventListenerBase::EventListenerBase;
```c++ void testRunStarting(Catch::TestRunInfo const&) override {
#define CATCH_CONFIG_MAIN lib_foo_init();
#include "catch.hpp"
struct MyListener : Catch::TestEventListenerBase {
using TestEventListenerBase::TestEventListenerBase; // inherit constructor
void testCaseStarting( Catch::TestCaseInfo const& testInfo ) override {
// Perform some setup before a test case is run
}
void testCaseEnded( Catch::TestCaseStats const& testCaseStats ) override {
// Tear-down after a test case is run
} }
}; };
CATCH_REGISTER_LISTENER( MyListener )
CATCH_REGISTER_LISTENER(testRunListener)
``` ```
_Note that you should not use any assertion macros within a Listener!_ _Note that you should not use any assertion macros within a Listener!_
## Events that can be hooked [You can find the list of events that the listeners can react to on its
own page](reporter-events.md#top).
The following are the methods that can be overridden in the Listener:
```c++
// The whole test run, starting and ending
virtual void testRunStarting( TestRunInfo const& testRunInfo );
virtual void testRunEnded( TestRunStats const& testRunStats );
// Test cases starting and ending
virtual void testCaseStarting( TestCaseInfo const& testInfo );
virtual void testCaseEnded( TestCaseStats const& testCaseStats );
// Sections starting and ending
virtual void sectionStarting( SectionInfo const& sectionInfo );
virtual void sectionEnded( SectionStats const& sectionStats );
// Assertions before/ after
virtual void assertionStarting( AssertionInfo const& assertionInfo );
virtual bool assertionEnded( AssertionStats const& assertionStats );
// A test is being skipped (because it is "hidden")
virtual void skipTest( TestCaseInfo const& testInfo );
```
More information about the events (e.g. name of the test case) is contained in the structs passed as arguments -
just look in the source code to see what fields are available.
--- ---

175
docs/reporter-events.md Normal file
View File

@ -0,0 +1,175 @@
<a id="top"></a>
# Reporter events
**Contents**<br>
[Test running events](#test-running-events)<br>
[Benchmarking events](#benchmarking-events)<br>
[Listings events](#listings-events)<br>
[Miscellaneous events](#miscellaneous-events)<br>
Reporter events are one of the customization points for user code. They
are used by [reporters](reporters.md#top) to customize Catch2's output,
and by [event listeners](event-listeners.md#top) to perform in-process
actions under some conditions.
There are currently 21 reporter events in Catch2, split between 4 distinct
event groups:
* test running events (10 events)
* benchmarking (4 events)
* listings (3 events)
* miscellaneous (4 events)
## Test running events
Test running events are always paired so that for each `fooStarting` event,
there is a `fooEnded` event. This means that the 10 test running events
consist of 5 pairs of events:
* `testRunStarting` and `testRunEnded`,
* `testCaseStarting` and `testCaseEnded`,
* `testCasePartialStarting` and `testCasePartialEnded`,
* `sectionStarting` and `sectionEnded`,
* `assertionStarting` and `assertionEnded`
### `testRun` events
```cpp
void testRunStarting( TestRunInfo const& testRunInfo );
void testRunEnded( TestRunStats const& testRunStats );
```
The `testRun` events bookend the entire test run. `testRunStarting` is
emitted before the first test case is executed, and `testRunEnded` is
emitted after all the test cases have been executed.
### `testCase` events
```cpp
void testCaseStarting( TestCaseInfo const& testInfo );
void testCaseEnded( TestCaseStats const& testCaseStats );
```
The `testCase` events bookend one _full_ run of a specific test case.
Individual runs through a test case, e.g. due to `SECTION`s or `GENERATE`s,
are handled by a different event.
### `testCasePartial` events
> Introduced in Catch2 X.Y.Z
```cpp
void testCasePartialStarting( TestCaseInfo const& testInfo, uint64_t partNumber );
void testCasePartialEnded(TestCaseStats const& testCaseStats, uint64_t partNumber );
```
`testCasePartial` events bookend one _partial_ run of a specific test case.
This means that for any given test case, these events can be emitted
multiple times, e.g. due to multiple leaf sections.
In regards to nesting with `testCase` events, `testCasePartialStarting`
will never be emitted before the corresponding `testCaseStarting`, and
`testCasePartialEnded` will always be emitted before the corresponding
`testCaseEnded`.
### `section` events
```cpp
void sectionStarting( SectionInfo const& sectionInfo );
void sectionEnded( SectionStats const& sectionStats );
```
`section` events are emitted only for active `SECTION`s, that is, sections
that are entered. Sections that are skipped in this test case run-through
do not cause events to be emitted.
_Note that test cases always contain one implicit section. The event for
this section is emitted after the corresponding `testCasePartialStarting`
event._
### `assertion` events
```cpp
void assertionStarting( AssertionInfo const& assertionInfo );
void assertionEnded( AssertionStats const& assertionStats );
```
`assertionStarting` is called after the expression is captured, but before
the assertion expression is evaluated. This might seem like a minor
distinction, but what it means is that if you have assertion like
`REQUIRE( a + b == c + d )`, then what happens is that `a + b` and `c + d`
are evaluated before `assertionStarting` is emitted, while the `==` is
evaluated after the event.
## Benchmarking events
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
```cpp
void benchmarkPreparing( StringRef name ) override;
void benchmarkStarting( BenchmarkInfo const& benchmarkInfo ) override;
void benchmarkEnded( BenchmarkStats<> const& benchmarkStats ) override;
void benchmarkFailed( StringRef error ) override;
```
Due to the benchmark lifecycle being bit more complicated, the benchmarking
events have their own category, even though they could be seen as parallel
to the `assertion*` events. You should expect running a benchmark to
generate at least 2 of the events above.
To understand the explanation below, you should read the [benchmarking
documentation](benchmarks.md#top) first.
* `benchmarkPreparing` event is sent after the environmental probe
finishes, but before the user code is first estimated.
* `benchmarkStarting` event is sent after the user code is estimated,
but has not been benchmarked yet.
* `benchmarkEnded` event is sent after the user code has been benchmarked,
and contains the benchmarking results.
* `benchmarkFailed` event is sent if either the estimation or the
benchmarking itself fails.
## Listings events
> Introduced in Catch2 X.Y.Z.
Listings events are events that correspond to the test binary being
invoked with `--list-foo` flag.
There are currently 3 listing events, one for reporters, one for tests,
and one for tags. Note that they are not exclusive to each other.
```cpp
void listReporters( std::vector<ReporterDescription> const& descriptions );
void listTests( std::vector<TestCaseHandle> const& tests );
void listTags( std::vector<TagInfo> const& tagInfos );
```
## Miscellaneous events
```cpp
void reportInvalidArguments( StringRef unmatchedSpec );
void fatalErrorEncountered( StringRef error );
void noMatchingTestCases( StringRef unmatchedSpec );
```
These are one-off events that do not neatly fit into other categories.
`reportInvalidArguments` is sent for each [test specification command line
argument](command-line.md#specifying-which-tests-to-run) that wasn't
parsed into a valid spec.
`fatalErrorEncountered` is sent when Catch2's POSIX signal handling
or Windows SE handler is called into with a fatal signal/exception.
`noMatchingTestCases` is sent for each user provided test specification
that did not match any registered tests.
---
[Home](Readme.md#top)