Add support for multiple parallel reporters

This requires a bunch of different changes across the reporter
subsystem.

* We need to handle multiple reporters and their differing
  preferences in `ListeningReporter`, e.g. what to do when
  we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
  how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
This commit is contained in:
Martin Jeřábek
2021-02-06 20:12:07 +01:00
committed by Martin Hořeňovský
parent 6b55f5d780
commit ccd67b293d
33 changed files with 51234 additions and 224 deletions

View File

@@ -123,7 +123,7 @@ Test names containing special characters, such as `,` or `[` can specify them on
<a id="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use
<pre>-r, --reporter &lt;reporter></pre>
<pre>-r, --reporter &lt;reporter[::output-file]&gt;</pre>
A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.<br />
The bundled reporters are:
@@ -136,6 +136,13 @@ The bundled reporters are:
The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Jenkins. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node.
This option may be passed multiple times to use multiple (different) reporters at the same time. See [Reporters](reporters.md#multiple-reporters) for details.
_Note: There is currently no way to escape `::` in the reporter spec,
and thus reporter/file names with `::` in them will not work properly.
As `::` in paths is relatively obscure (unlike `:`), we do not consider
this an issue._
<a id="breaking-into-the-debugger"></a>
## Breaking into the debugger
<pre>-b, --break</pre>
@@ -178,7 +185,7 @@ If one or more test-specs have been supplied too then only the matching tests wi
<a id="sending-output-to-a-file"></a>
## Sending output to a file
<pre>-o, --out &lt;filename>
<pre>-o, --out &lt;filename&gt;
</pre>
Use this option to send all output to a file. By default output is sent to stdout (note that uses of stdout and stderr *from within test cases* are redirected and included in the report - so even stderr will effectively end up on stdout).
@@ -414,7 +421,7 @@ There are some limitations of this feature to be aware of:
- Code outside of sections being skipped will still be executed - e.g. any set-up code in the TEST_CASE before the
start of the first section.</br>
- At time of writing, wildcards are not supported in section names.
- If you specify a section without narrowing to a test case first then all test cases will be executed
- If you specify a section without narrowing to a test case first then all test cases will be executed
(but only matching sections within them).
@@ -422,7 +429,7 @@ start of the first section.</br>
## Filenames as tags
<pre>-#, --filenames-as-tags</pre>
When this option is used then every test is given an additional tag which is formed of the unqualified
When this option is used then every test is given an additional tag which is formed of the unqualified
filename it is found in, with any extension stripped, prefixed with the `#` character.
So, for example, tests within the file `~\Dev\MyProject\Ferrets.cpp` would be tagged `[#Ferrets]`.

View File

@@ -19,13 +19,13 @@ There are four reporters built in to the single include:
* `console` writes as lines of text, formatted to a typical terminal width, with colours if a capable terminal is detected.
* `compact` similar to `console` but optimised for minimal output - each entry on one line
* `junit` writes xml that corresponds to Ant's [junitreport](http://help.catchsoftware.com/display/ET/JUnit+Format) target. Useful for build systems that understand Junit.
Because of the way the junit format is structured the run must complete before anything is written.
Because of the way the junit format is structured the run must complete before anything is written.
* `xml` writes an xml format tailored to Catch. Unlike `junit` this is a streaming format so results are delivered progressively.
There are a few additional reporters, for specific build systems, in the Catch repository (in `include\reporters`) which you can `#include` in your project if you would like to make use of them.
Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`.
* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)).
* `tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format.
* `automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files
@@ -35,6 +35,17 @@ You see what reporters are available from the command line by running with `--li
By default all these reports are written to stdout, but can be redirected to a file with [`-o` or `--out`](command-line.md#sending-output-to-a-file)
<a id="multiple-reporters"></a>
## Using multiple reporters
Multiple reporters may be used at the same time, e.g. to save a machine-readable output to a file but still print the human-readable output to the console:
```
-r console -r xml::result.xml -r junit::result-junit.xml
```
The output file name is given after the reporter name, delimited by a colon. If omitted, it defaults to the file name specified by `-o` (or stdout). Only one reporter may use the default output.
## Writing your own reporter
You can write your own custom reporter and register it with Catch.