diff --git a/docs/event-listeners.md b/docs/event-listeners.md index 62311006..2dbd2a81 100644 --- a/docs/event-listeners.md +++ b/docs/event-listeners.md @@ -1,74 +1,43 @@ # Event Listeners -A `Listener` is a class you can register with Catch that will then be passed events, -such as a test case starting or ending, as they happen during a test run. -`Listeners` are actually types of `Reporters`, with a few small differences: - -1. Once registered in code they are automatically used - you don't need to specify them on the command line -2. They are called in addition to (just before) any reporters, and you can register multiple listeners. -3. They derive from `Catch::TestEventListenerBase`, which has default stubs for all the events, -so you are not forced to implement events you're not interested in. -4. You register a listener with `CATCH_REGISTER_LISTENER` +An event listener is a bit like a reporter, in that it responds to various +reporter events in Catch2, but it is not expected to write any output. +Instead, an event listener performs actions within the test process, such +as performing global initialization (e.g. of a C library), or cleaning out +in-memory logs if they are not needed (the test case passed). +Unlike reporters, each registered event listener is always active. Event +listeners are always notified before reporter(s). -## Implementing a Listener -Simply derive a class from `Catch::TestEventListenerBase` and implement the methods you are interested in, either in -the main source file (i.e. the one that defines `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`), or in a -file that defines `CATCH_CONFIG_EXTERNAL_INTERFACES`. +To write your own event listener, you should derive from `Catch::TestEventListenerBase`, +as it provides empty stubs for all reporter events, allowing you to +only override events you care for. Afterwards you have to register it +with Catch2 using `CATCH_REGISTER_LISTENER` macro, so that Catch2 knows +about it and instantiates it before running tests. -Then register it using `CATCH_REGISTER_LISTENER`. +Example event listener: +```cpp +#include +#include -For example ([complete source code](../examples/210-Evt-EventListeners.cpp)): +class testRunListener : public Catch::EventListenerBase { +public: + using Catch::EventListenerBase::EventListenerBase; -```c++ -#define CATCH_CONFIG_MAIN -#include "catch.hpp" - -struct MyListener : Catch::TestEventListenerBase { - - using TestEventListenerBase::TestEventListenerBase; // inherit constructor - - void testCaseStarting( Catch::TestCaseInfo const& testInfo ) override { - // Perform some setup before a test case is run - } - - void testCaseEnded( Catch::TestCaseStats const& testCaseStats ) override { - // Tear-down after a test case is run + void testRunStarting(Catch::TestRunInfo const&) override { + lib_foo_init(); } }; -CATCH_REGISTER_LISTENER( MyListener ) + +CATCH_REGISTER_LISTENER(testRunListener) ``` _Note that you should not use any assertion macros within a Listener!_ -## Events that can be hooked +[You can find the list of events that the listeners can react to on its +own page](reporter-events.md#top). -The following are the methods that can be overridden in the Listener: - -```c++ -// The whole test run, starting and ending -virtual void testRunStarting( TestRunInfo const& testRunInfo ); -virtual void testRunEnded( TestRunStats const& testRunStats ); - -// Test cases starting and ending -virtual void testCaseStarting( TestCaseInfo const& testInfo ); -virtual void testCaseEnded( TestCaseStats const& testCaseStats ); - -// Sections starting and ending -virtual void sectionStarting( SectionInfo const& sectionInfo ); -virtual void sectionEnded( SectionStats const& sectionStats ); - -// Assertions before/ after -virtual void assertionStarting( AssertionInfo const& assertionInfo ); -virtual bool assertionEnded( AssertionStats const& assertionStats ); - -// A test is being skipped (because it is "hidden") -virtual void skipTest( TestCaseInfo const& testInfo ); -``` - -More information about the events (e.g. name of the test case) is contained in the structs passed as arguments - -just look in the source code to see what fields are available. --- diff --git a/docs/reporter-events.md b/docs/reporter-events.md new file mode 100644 index 00000000..b5374460 --- /dev/null +++ b/docs/reporter-events.md @@ -0,0 +1,175 @@ + +# Reporter events + +**Contents**
+[Test running events](#test-running-events)
+[Benchmarking events](#benchmarking-events)
+[Listings events](#listings-events)
+[Miscellaneous events](#miscellaneous-events)
+ +Reporter events are one of the customization points for user code. They +are used by [reporters](reporters.md#top) to customize Catch2's output, +and by [event listeners](event-listeners.md#top) to perform in-process +actions under some conditions. + +There are currently 21 reporter events in Catch2, split between 4 distinct +event groups: +* test running events (10 events) +* benchmarking (4 events) +* listings (3 events) +* miscellaneous (4 events) + +## Test running events + +Test running events are always paired so that for each `fooStarting` event, +there is a `fooEnded` event. This means that the 10 test running events +consist of 5 pairs of events: + +* `testRunStarting` and `testRunEnded`, +* `testCaseStarting` and `testCaseEnded`, +* `testCasePartialStarting` and `testCasePartialEnded`, +* `sectionStarting` and `sectionEnded`, +* `assertionStarting` and `assertionEnded` + +### `testRun` events + +```cpp +void testRunStarting( TestRunInfo const& testRunInfo ); +void testRunEnded( TestRunStats const& testRunStats ); +``` + +The `testRun` events bookend the entire test run. `testRunStarting` is +emitted before the first test case is executed, and `testRunEnded` is +emitted after all the test cases have been executed. + +### `testCase` events + +```cpp +void testCaseStarting( TestCaseInfo const& testInfo ); +void testCaseEnded( TestCaseStats const& testCaseStats ); +``` + +The `testCase` events bookend one _full_ run of a specific test case. +Individual runs through a test case, e.g. due to `SECTION`s or `GENERATE`s, +are handled by a different event. + + +### `testCasePartial` events + +> Introduced in Catch2 X.Y.Z + +```cpp +void testCasePartialStarting( TestCaseInfo const& testInfo, uint64_t partNumber ); +void testCasePartialEnded(TestCaseStats const& testCaseStats, uint64_t partNumber ); +``` + +`testCasePartial` events bookend one _partial_ run of a specific test case. +This means that for any given test case, these events can be emitted +multiple times, e.g. due to multiple leaf sections. + +In regards to nesting with `testCase` events, `testCasePartialStarting` +will never be emitted before the corresponding `testCaseStarting`, and +`testCasePartialEnded` will always be emitted before the corresponding +`testCaseEnded`. + + +### `section` events + +```cpp +void sectionStarting( SectionInfo const& sectionInfo ); +void sectionEnded( SectionStats const& sectionStats ); +``` + +`section` events are emitted only for active `SECTION`s, that is, sections +that are entered. Sections that are skipped in this test case run-through +do not cause events to be emitted. + +_Note that test cases always contain one implicit section. The event for +this section is emitted after the corresponding `testCasePartialStarting` +event._ + + +### `assertion` events + +```cpp +void assertionStarting( AssertionInfo const& assertionInfo ); +void assertionEnded( AssertionStats const& assertionStats ); +``` + +`assertionStarting` is called after the expression is captured, but before +the assertion expression is evaluated. This might seem like a minor +distinction, but what it means is that if you have assertion like +`REQUIRE( a + b == c + d )`, then what happens is that `a + b` and `c + d` +are evaluated before `assertionStarting` is emitted, while the `==` is +evaluated after the event. + + +## Benchmarking events + +> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0. + +```cpp +void benchmarkPreparing( StringRef name ) override; +void benchmarkStarting( BenchmarkInfo const& benchmarkInfo ) override; +void benchmarkEnded( BenchmarkStats<> const& benchmarkStats ) override; +void benchmarkFailed( StringRef error ) override; +``` + +Due to the benchmark lifecycle being bit more complicated, the benchmarking +events have their own category, even though they could be seen as parallel +to the `assertion*` events. You should expect running a benchmark to +generate at least 2 of the events above. + +To understand the explanation below, you should read the [benchmarking +documentation](benchmarks.md#top) first. + +* `benchmarkPreparing` event is sent after the environmental probe +finishes, but before the user code is first estimated. +* `benchmarkStarting` event is sent after the user code is estimated, +but has not been benchmarked yet. +* `benchmarkEnded` event is sent after the user code has been benchmarked, +and contains the benchmarking results. +* `benchmarkFailed` event is sent if either the estimation or the +benchmarking itself fails. + + +## Listings events + +> Introduced in Catch2 X.Y.Z. + +Listings events are events that correspond to the test binary being +invoked with `--list-foo` flag. + +There are currently 3 listing events, one for reporters, one for tests, +and one for tags. Note that they are not exclusive to each other. + +```cpp +void listReporters( std::vector const& descriptions ); +void listTests( std::vector const& tests ); +void listTags( std::vector const& tagInfos ); +``` + + +## Miscellaneous events + +```cpp +void reportInvalidArguments( StringRef unmatchedSpec ); +void fatalErrorEncountered( StringRef error ); +void noMatchingTestCases( StringRef unmatchedSpec ); +``` + +These are one-off events that do not neatly fit into other categories. + +`reportInvalidArguments` is sent for each [test specification command line +argument](command-line.md#specifying-which-tests-to-run) that wasn't +parsed into a valid spec. + +`fatalErrorEncountered` is sent when Catch2's POSIX signal handling +or Windows SE handler is called into with a fatal signal/exception. + +`noMatchingTestCases` is sent for each user provided test specification +that did not match any registered tests. + +--- + +[Home](Readme.md#top)