mirror of
https://github.com/catchorg/Catch2.git
synced 2024-12-22 19:33:29 +01:00
Martin Hořeňovský
parent
eeee4a49af
commit
d3199c42c2
@ -30,7 +30,7 @@
|
||||
[Filenames as tags](#filenames-as-tags)<br>
|
||||
[Override output colouring](#override-output-colouring)<br>
|
||||
[Test Sharding](#test-sharding)<br>
|
||||
[Allow running the binary without tests](#no-tests-override)<br>
|
||||
[Allow running the binary without tests](#allow-running-the-binary-without-tests)<br>
|
||||
|
||||
Catch works quite nicely without any command line options at all - but for those times when you want greater control the following options are available.
|
||||
Click one of the following links to take you straight to that option - or scroll on to browse the available options.
|
||||
@ -127,20 +127,29 @@ Test names containing special characters, such as `,` or `[` can specify them on
|
||||
|
||||
> Support for providing output-file through the `-r`, `--reporter` flag was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
|
||||
|
||||
A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.<br />
|
||||
The bundled reporters are:
|
||||
Reporters are how the output from Catch2 (results of assertions, tests,
|
||||
benchmarks and so on) is formatted and written out. The default reporter
|
||||
is called the "Console" reporter and is intended to provide relatively
|
||||
verbose and human-friendly output.
|
||||
|
||||
<pre>-r console
|
||||
-r compact
|
||||
-r xml
|
||||
-r junit
|
||||
</pre>
|
||||
There are multiple built-in reporters, you can see what they do by using the
|
||||
[`--list-reporter`](command-line.md#listing-available-tests-tags-or-reporters)
|
||||
flag. If you need a reporter providing custom format outside of the already
|
||||
provided ones, look at the ["write your own reporter" part of the reporter
|
||||
documentation](reporters.md#writing-your-own-reporter).
|
||||
|
||||
The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Jenkins. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node.
|
||||
|
||||
This option may be passed multiple times to use multiple (different) reporters at the same time. See [Reporters](reporters.md#multiple-reporters) for details.
|
||||
This option may be passed multiple times to use multiple (different)
|
||||
reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters)
|
||||
for details on what the resulting behaviour is. Also note that at most one
|
||||
reporter can be provided without the output-file part of reporter spec.
|
||||
This reporter will use the "default" output destination, based on
|
||||
the [`-o`, `--out`](#sending-output-to-a-file) option.
|
||||
|
||||
As with the `--out` flag, `-` means writing to stdout.
|
||||
> Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
|
||||
|
||||
As with the `--out` option, using `-` for the output file name sends the
|
||||
output to stdout.
|
||||
|
||||
_Note: There is currently no way to escape `::` in the reporter spec,
|
||||
and thus reporter/file names with `::` in them will not work properly.
|
||||
|
||||
@ -1,59 +1,187 @@
|
||||
<a id="top"></a>
|
||||
# Reporters
|
||||
|
||||
Catch has a modular reporting system and comes bundled with a handful of useful reporters built in.
|
||||
You can also write your own reporters.
|
||||
Reporters are a customization point for most of Catch2's output, e.g.
|
||||
formatting and writing out [assertions (whether passing or failing),
|
||||
sections, test cases, benchmarks, and so on](reporter-events.md#top).
|
||||
|
||||
Catch2 comes with a bunch of reporters by default (currently 8), and
|
||||
you can also write your own reporter. Because multiple reporters can
|
||||
be active at the same time, your own reporters do not even have to handle
|
||||
all reporter event, just the ones you are interested in, e.g. benchmarks.
|
||||
|
||||
|
||||
## Using different reporters
|
||||
|
||||
The reporter to use can easily be controlled from the command line.
|
||||
To specify a reporter use [`-r` or `--reporter`](command-line.md#choosing-a-reporter-to-use), followed by the name of the reporter, e.g.:
|
||||
You can see which reporters are available by running the test binary
|
||||
with `--list-reporters`. You can then pick one of them with the [`-r`,
|
||||
`--reporter` option](command-line.md#choosing-a-reporter-to-use), followed
|
||||
by the name of the desired reporter, like so:
|
||||
|
||||
```
|
||||
-r xml
|
||||
--reporter xml
|
||||
```
|
||||
|
||||
If you don't specify a reporter then the console reporter is used by default.
|
||||
There are four reporters built in to the single include:
|
||||
You can also select multiple reporters to be used at the same time.
|
||||
In that case you should read the [section on using multiple
|
||||
reporters](#multiple-reporters) to avoid any surprises from doing so.
|
||||
|
||||
* `console` writes as lines of text, formatted to a typical terminal width, with colours if a capable terminal is detected.
|
||||
* `compact` similar to `console` but optimised for minimal output - each entry on one line
|
||||
* `junit` writes xml that corresponds to Ant's [junitreport](http://help.catchsoftware.com/display/ET/JUnit+Format) target. Useful for build systems that understand Junit.
|
||||
Because of the way the junit format is structured the run must complete before anything is written.
|
||||
* `xml` writes an xml format tailored to Catch. Unlike `junit` this is a streaming format so results are delivered progressively.
|
||||
|
||||
There are a few additional reporters, for specific build systems, in the Catch repository (in `include\reporters`) which you can `#include` in your project if you would like to make use of them.
|
||||
Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`.
|
||||
|
||||
* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
|
||||
Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)).
|
||||
* `tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format.
|
||||
* `automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files
|
||||
* `sonarqube` writes the [SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format.
|
||||
|
||||
You see what reporters are available from the command line by running with `--list-reporters`.
|
||||
|
||||
By default all these reports are written to stdout, but can be redirected to a file with [`-o` or `--out`](command-line.md#sending-output-to-a-file)
|
||||
|
||||
<a id="multiple-reporters"></a>
|
||||
## Using multiple reporters
|
||||
|
||||
> Support for having multiple parallel reporters was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
|
||||
|
||||
Multiple reporters may be used at the same time, e.g. to save a machine-readable output to a file but still print the human-readable output to the console:
|
||||
Catch2 supports using multiple reporters at the same time while having
|
||||
them write into different destinations. The two main uses of this are
|
||||
|
||||
* having both human-friendly and machine-parseable (e.g. in JUnit format)
|
||||
output from one run of binary
|
||||
* having "partial" reporters that are highly specialized, e.g. having one
|
||||
reporter that writes out benchmark results as markdown tables and does
|
||||
nothing else, while also having standard testing output separately
|
||||
|
||||
Specifying multiple reporter looks like this:
|
||||
```
|
||||
-r console -r xml::result.xml -r junit::result-junit.xml
|
||||
--reporter console::- --reporter JUnit::result-junit.xml
|
||||
```
|
||||
|
||||
The output file name is given after the reporter name, delimited by a colon. If omitted, it defaults to the file name specified by `-o` (or stdout). Only one reporter may use the default output.
|
||||
Using multiple reporters (or one reporter and one-or-more [event
|
||||
listeners](event-listener.md#top)) can have surprisingly complex semantics
|
||||
when using customization points provided to reporters by Catch2, namely
|
||||
capturing stdout/stderr from test cases.
|
||||
|
||||
As long as at least one reporter (or listener) asks Catch2 to capture
|
||||
stdout/stderr, captured stdout and stderr will be available to all
|
||||
reporters and listeners.
|
||||
|
||||
Because this might be surprising to the users, if at least one active
|
||||
_reporter_ is non-capturing, then Catch2 tries to roughly emulate
|
||||
non-capturing behaviour by printing out the captured stdout/stderr
|
||||
just before `testCasePartialEnded` event is sent out to the active
|
||||
reporters and listeners. This means that stdout/stderr is no longer
|
||||
printed out from tests as it is being written, but instead it is written
|
||||
out in batch after each runthrough of a test case is finished.
|
||||
|
||||
|
||||
|
||||
## Writing your own reporter
|
||||
|
||||
You can write your own custom reporter and register it with Catch.
|
||||
At time of writing the interface is subject to some changes so is not, yet, documented here.
|
||||
If you are determined you shouldn't have too much trouble working it out from the existing implementations -
|
||||
but do keep in mind upcoming changes (these will be minor, simplifying, changes such as not needing to forward calls to the base class).
|
||||
You can also write your own custom reporter and tell Catch2 to use it.
|
||||
When writing your reporter, you have two options:
|
||||
|
||||
* Derive from `Catch::IStreamingReporter`. When doing this, you will have
|
||||
to provide handling for all [reporter events](reporter-events.md#top).
|
||||
* Derive from one of the provided [utility reporter bases in
|
||||
Catch2](#utility-reporter-bases).
|
||||
|
||||
Generally we recommend doing the latter, as it is less work.
|
||||
|
||||
Apart from overriding handling of the individual reporter events, reporters
|
||||
have access to some extra customization points, described below.
|
||||
|
||||
|
||||
### Utility reporter bases
|
||||
|
||||
Catch2 currently provides two utility reporter bases:
|
||||
|
||||
* `Catch::StreamingReporterBase`
|
||||
* `Catch::CumulativeReporterBase`
|
||||
|
||||
`StreamingReporterBase` is useful for reporters that can format and write
|
||||
out the events as they come in. It provides (usually empty) implementation
|
||||
for all reporter events, and if you let it handle the relevant events,
|
||||
it also handles storing information about active test run and test case.
|
||||
|
||||
`CumulativeReporterBase` is a base for reporters that need to see the whole
|
||||
test run, before they can start writing the output, such as the JUnit
|
||||
and SonarQube reporters. This post-facto approach requires the assertions
|
||||
to be stringified when it is finished, so that the assertion can be written
|
||||
out later. Because the stringification can be expensive, and not all
|
||||
cumulative reporters need the assertions, this base provides customization
|
||||
point to change whether the assertions are saved or not, separate for
|
||||
passing and failing assertions.
|
||||
|
||||
|
||||
_Generally we recommend that if you override a member function from either
|
||||
of the bases, you call into the base's implementation first. This is not
|
||||
necessarily in all cases, but it is safer and easier._
|
||||
|
||||
|
||||
Writing your own reporter then looks like this:
|
||||
|
||||
```cpp
|
||||
#include <catch2/reporters/catch_reporter_streaming_base.hpp>
|
||||
#include <catch2/catch_test_case_info.hpp>
|
||||
#include <catch2/reporters/catch_reporter_registrars.hpp>
|
||||
|
||||
#include <iostream>
|
||||
|
||||
class PartialReporter : public Catch::StreamingReporterBase {
|
||||
public:
|
||||
using StreamingReporterBase::StreamingReporterBase;
|
||||
|
||||
static std::string getDescription() {
|
||||
return "Reporter for testing TestCasePartialStarting/Ended events";
|
||||
}
|
||||
|
||||
void testCasePartialStarting(Catch::TestCaseInfo const& testInfo,
|
||||
uint64_t partNumber) override {
|
||||
std::cout << "TestCaseStartingPartial: " << testInfo.name << '#' << partNumber << '\n';
|
||||
}
|
||||
|
||||
void testCasePartialEnded(Catch::TestCaseStats const& testCaseStats,
|
||||
uint64_t partNumber) override {
|
||||
std::cout << "TestCasePartialEnded: " << testCaseStats.testInfo->name << '#' << partNumber << '\n';
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
CATCH_REGISTER_REPORTER("partial", PartialReporter)
|
||||
```
|
||||
|
||||
This create a simple reporter that responds to `testCasePartial*` events,
|
||||
and calls itself "partial" reporter, so it can be invoked with
|
||||
`--reporter partial` command line flag.
|
||||
|
||||
|
||||
### `ReporterPreferences`
|
||||
|
||||
Each reporter instance contains instance of `ReporterPreferences`, a type
|
||||
that holds flags for the behaviour of Catch2 when this reporter run.
|
||||
Currently there are two customization options:
|
||||
|
||||
* `shouldRedirectStdOut` - whether the reporter wants to handle
|
||||
writes to stdout/stderr from user code, or not. This is useful for
|
||||
reporters that output machine-parseable output, e.g. the JUnit
|
||||
reporter, or the XML reporter.
|
||||
* `shouldReportAllAssertions` - whether the reporter wants to handle
|
||||
`assertionEnded` events for passing assertions as well as failing
|
||||
assertions. Usually reporters do not report successful assertions
|
||||
and don't need them for their output, but sometimes the desired output
|
||||
format includes passing assertions even without the `-s` flag.
|
||||
|
||||
|
||||
### Other expected functionality of a reporter
|
||||
|
||||
When writing a custom reporter, there are few more things that you should
|
||||
keep in mind. These are not important for correctness, but they are
|
||||
important for the reporter to work _nicely_.
|
||||
|
||||
* Catch2 provides a simple verbosity option for users. There are three
|
||||
verbosity levels, "quiet", "normal", and "high", and if it makes sense
|
||||
for reporter's output format, it should respond to these by changing
|
||||
what, and how much, it writes out.
|
||||
|
||||
* Catch2 operates with an rng-seed. Knowing what seed a test run had
|
||||
is important if you want to replicate it, so your reporter should
|
||||
report the rng-seed, if at all possible given the target output format.
|
||||
|
||||
* Catch2 also operates with test filters, or test specs. If a filter
|
||||
is present, you should also report the filter, if at all possible given
|
||||
the target output format.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user