3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-12-22 11:23:29 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add support for multiple parallel reporters

Browse Source 
This requires a bunch of different changes across the reporter
subsystem.

* We need to handle multiple reporters and their differing
  preferences in `ListeningReporter`, e.g. what to do when
  we mix reporters that capture and don't capture stdout.
* We need to change how the reporter is given output and
  how we parse reporter's output destination from CLI.
* Approval tests need to handle multireporter option
This commit is contained in:
Martin Jeřábek 2021-02-06 20:12:07 +01:00 committed by Martin Hořeňovský
parent 6b55f5d780
commit ccd67b293d
33 changed files with 51234 additions and 224 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/command-line.md
View File

@ -123,7 +123,7 @@ Test names containing special characters, such as `,` or `[` can specify them on
<a id="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use
<pre>-r, --reporter &lt;reporter></pre>
<pre>-r, --reporter &lt;reporter[::output-file]&gt;</pre>
A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.<br />
The bundled reporters are:
@ -136,6 +136,13 @@ The bundled reporters are:
The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Jenkins. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node.
This option may be passed multiple times to use multiple (different) reporters at the same time. See [Reporters](reporters.md#multiple-reporters) for details.
_Note: There is currently no way to escape `::` in the reporter spec,
and thus reporter/file names with `::` in them will not work properly.
As `::` in paths is relatively obscure (unlike `:`), we do not consider
this an issue._
<a id="breaking-into-the-debugger"></a>
## Breaking into the debugger
<pre>-b, --break</pre>
@ -178,7 +185,7 @@ If one or more test-specs have been supplied too then only the matching tests wi
<a id="sending-output-to-a-file"></a>
## Sending output to a file
<pre>-o, --out &lt;filename>
<pre>-o, --out &lt;filename&gt;
</pre>
Use this option to send all output to a file. By default output is sent to stdout (note that uses of stdout and stderr *from within test cases* are redirected and included in the report - so even stderr will effectively end up on stdout).
@ -414,7 +421,7 @@ There are some limitations of this feature to be aware of:
- Code outside of sections being skipped will still be executed - e.g. any set-up code in the TEST_CASE before the
start of the first section.</br>
- At time of writing, wildcards are not supported in section names.
- If you specify a section without narrowing to a test case first then all test cases will be executed
- If you specify a section without narrowing to a test case first then all test cases will be executed
(but only matching sections within them).
@ -422,7 +429,7 @@ start of the first section.</br>
## Filenames as tags
<pre>-#, --filenames-as-tags</pre>
When this option is used then every test is given an additional tag which is formed of the unqualified
When this option is used then every test is given an additional tag which is formed of the unqualified
filename it is found in, with any extension stripped, prefixed with the `#` character.
So, for example, tests within the file `~\Dev\MyProject\Ferrets.cpp` would be tagged `[#Ferrets]`.

15
docs/reporters.md
View File

@ -19,13 +19,13 @@ There are four reporters built in to the single include:
* `console` writes as lines of text, formatted to a typical terminal width, with colours if a capable terminal is detected.
* `compact` similar to `console` but optimised for minimal output - each entry on one line
* `junit` writes xml that corresponds to Ant's [junitreport](http://help.catchsoftware.com/display/ET/JUnit+Format) target. Useful for build systems that understand Junit.
Because of the way the junit format is structured the run must complete before anything is written.
Because of the way the junit format is structured the run must complete before anything is written.
* `xml` writes an xml format tailored to Catch. Unlike `junit` this is a streaming format so results are delivered progressively.
There are a few additional reporters, for specific build systems, in the Catch repository (in `include\reporters`) which you can `#include` in your project if you would like to make use of them.
Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`.
* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)).
* `tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format.
* `automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files
@ -35,6 +35,17 @@ You see what reporters are available from the command line by running with `--li
By default all these reports are written to stdout, but can be redirected to a file with [`-o` or `--out`](command-line.md#sending-output-to-a-file)
<a id="multiple-reporters"></a>
## Using multiple reporters
Multiple reporters may be used at the same time, e.g. to save a machine-readable output to a file but still print the human-readable output to the console:
```
-r console -r xml::result.xml -r junit::result-junit.xml
```
The output file name is given after the reporter name, delimited by a colon. If omitted, it defaults to the file name specified by `-o` (or stdout). Only one reporter may use the default output.
## Writing your own reporter
You can write your own custom reporter and register it with Catch.

67
src/catch2/catch_config.cpp
View File

@ -13,13 +13,39 @@
#include <catch2/internal/catch_test_spec_parser.hpp>
#include <catch2/interfaces/catch_interfaces_tag_alias_registry.hpp>
#include <ostream>
namespace Catch {
namespace Detail {
namespace {
class RDBufStream : public IStream {
mutable std::ostream m_os;
Config::Config( ConfigData const& data )
: m_data( data ),
m_stream( Catch::makeStream(m_data.outputFilename) )
{
public:
//! The streambuf `sb` must outlive the constructed object.
RDBufStream( std::streambuf* sb ): m_os( sb ) {}
~RDBufStream() override = default;
public: // IStream
std::ostream& stream() const override { return m_os; }
};
} // unnamed namespace
} // namespace Detail
std::ostream& operator<<( std::ostream& os,
ConfigData::ReporterAndFile const& reporter ) {
os << "{ " << reporter.reporterName << ", ";
if ( reporter.outputFileName ) {
os << *reporter.outputFileName;
} else {
os << "<default-output>";
}
return os << " }";
}
Config::Config( ConfigData const& data ):
m_data( data ),
m_defaultStream( openStream( data.defaultOutputFilename ) ) {
// We need to trim filter specs to avoid trouble with superfluous
// whitespace (esp. important for bdd macros, as those are manually
// aligned with whitespace).
@ -39,24 +65,37 @@ namespace Catch {
}
}
m_testSpec = parser.testSpec();
m_reporterStreams.reserve( m_data.reporterSpecifications.size() );
for ( auto const& reporterAndFile : m_data.reporterSpecifications ) {
if ( reporterAndFile.outputFileName.none() ) {
m_reporterStreams.emplace_back( new Detail::RDBufStream(
m_defaultStream->stream().rdbuf() ) );
} else {
m_reporterStreams.emplace_back(
openStream( *reporterAndFile.outputFileName ) );
}
}
}
Config::~Config() = default;
std::string const& Config::getFilename() const {
return m_data.outputFilename ;
}
bool Config::listTests() const { return m_data.listTests; }
bool Config::listTags() const { return m_data.listTags; }
bool Config::listReporters() const { return m_data.listReporters; }
std::string const& Config::getReporterName() const { return m_data.reporterName; }
std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; }
std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; }
std::vector<ConfigData::ReporterAndFile> const& Config::getReportersAndOutputFiles() const {
return m_data.reporterSpecifications;
}
std::ostream& Config::getReporterOutputStream(std::size_t reporterIdx) const {
return m_reporterStreams.at(reporterIdx)->stream();
}
TestSpec const& Config::testSpec() const { return m_testSpec; }
bool Config::hasTestFilters() const { return m_hasTestFilters; }
@ -64,8 +103,8 @@ namespace Catch {
// IConfig interface
bool Config::allowThrows() const { return !m_data.noThrow; }
std::ostream& Config::stream() const { return m_stream->stream(); }
StringRef Config::name() const { return m_data.name.empty() ? m_data.processName : m_data.name; }
std::ostream& Config::defaultStream() const { return m_defaultStream->stream(); }
StringRef Config::name() const { return m_data.name.empty() ? m_data.processName : m_data.name; }
bool Config::includeSuccessfulResults() const { return m_data.showSuccessfulTests; }
bool Config::warnAboutMissingAssertions() const {
return !!( m_data.warnings & WarnAbout::NoAssertions );
@ -92,4 +131,8 @@ namespace Catch {
unsigned int Config::benchmarkResamples() const { return m_data.benchmarkResamples; }
std::chrono::milliseconds Config::benchmarkWarmupTime() const { return std::chrono::milliseconds(m_data.benchmarkWarmupTime); }
Detail::unique_ptr<IStream const> Config::openStream(std::string const& outputFileName) {
return Catch::makeStream(outputFileName);
}
} // end namespace Catch

33
src/catch2/catch_config.hpp
View File

@ -14,6 +14,7 @@
#include <catch2/internal/catch_optional.hpp>
#include <catch2/internal/catch_random_seed_generation.hpp>
#include <iosfwd>
#include <vector>
#include <string>
@ -22,6 +23,18 @@ namespace Catch {
struct IStream;
struct ConfigData {
struct ReporterAndFile {
std::string reporterName;
// If none, the output goes to the default output.
Optional<std::string> outputFileName;
friend bool operator==(ReporterAndFile const& lhs, ReporterAndFile const& rhs) {
return lhs.reporterName == rhs.reporterName && lhs.outputFileName == rhs.outputFileName;
}
friend std::ostream& operator<<(std::ostream &os, ReporterAndFile const& reporter);
};
bool listTests = false;
bool listTags = false;
bool listReporters = false;
@ -55,13 +68,17 @@ namespace Catch {
UseColour useColour = UseColour::Auto;
WaitForKeypress::When waitForKeypress = WaitForKeypress::Never;
std::string outputFilename;
std::string defaultOutputFilename;
std::string name;
std::string processName;
#ifndef CATCH_CONFIG_DEFAULT_REPORTER
#define CATCH_CONFIG_DEFAULT_REPORTER "console"
#endif
std::string reporterName = CATCH_CONFIG_DEFAULT_REPORTER;
std::vector<ReporterAndFile> reporterSpecifications = {
{CATCH_CONFIG_DEFAULT_REPORTER, {}}
};
// Internal: used as parser state
bool _nonDefaultReporterSpecifications = false;
#undef CATCH_CONFIG_DEFAULT_REPORTER
std::vector<std::string> testsOrTags;
@ -76,13 +93,12 @@ namespace Catch {
Config( ConfigData const& data );
~Config() override; // = default in the cpp file
std::string const& getFilename() const;
bool listTests() const;
bool listTags() const;
bool listReporters() const;
std::string const& getReporterName() const;
std::vector<ConfigData::ReporterAndFile> const& getReportersAndOutputFiles() const;
std::ostream& getReporterOutputStream(std::size_t reporterIdx) const;
std::vector<std::string> const& getTestsOrTags() const override;
std::vector<std::string> const& getSectionsToRun() const override;
@ -94,7 +110,7 @@ namespace Catch {
// IConfig interface
bool allowThrows() const override;
std::ostream& stream() const override;
std::ostream& defaultStream() const override;
StringRef name() const override;
bool includeSuccessfulResults() const override;
bool warnAboutMissingAssertions() const override;
@ -118,13 +134,14 @@ namespace Catch {
std::chrono::milliseconds benchmarkWarmupTime() const override;
private:
Detail::unique_ptr<IStream const> openStream(std::string const& outputFileName);
ConfigData m_data;
Detail::unique_ptr<IStream const> m_stream;
Detail::unique_ptr<IStream const> m_defaultStream;
std::vector<Detail::unique_ptr<IStream const>> m_reporterStreams;
TestSpec m_testSpec;
bool m_hasTestFilters = false;
};
} // end namespace Catch
#endif // CATCH_CONFIG_HPP_INCLUDED

20
src/catch2/catch_session.cpp
View File

@ -34,7 +34,7 @@ namespace Catch {
namespace {
const int MaxExitCode = 255;
IStreamingReporterPtr createReporter(std::string const& reporterName, IConfig const* config) {
IStreamingReporterPtr createReporter(std::string const& reporterName, ReporterConfig const& config) {
auto reporter = Catch::getRegistryHub().getReporterRegistry().create(reporterName, config);
CATCH_ENFORCE(reporter, "No reporter registered with name: '" << reporterName << '\'');
@ -42,16 +42,26 @@ namespace Catch {
}
IStreamingReporterPtr makeReporter(Config const* config) {
if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty()) {
return createReporter(config->getReporterName(), config);
if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty()
&& config->getReportersAndOutputFiles().size() == 1) {
auto& stream = config->getReporterOutputStream(0);
return createReporter(config->getReportersAndOutputFiles()[0].reporterName, ReporterConfig(config, stream));
}
auto multi = Detail::make_unique<ListeningReporter>(config);
auto const& listeners = Catch::getRegistryHub().getReporterRegistry().getListeners();
for (auto const& listener : listeners) {
multi->addListener(listener->create(Catch::ReporterConfig(config)));
multi->addListener(listener->create(Catch::ReporterConfig(config, config->defaultStream())));
}
multi->addReporter(createReporter(config->getReporterName(), config));
std::size_t reporterIdx = 0;
for (auto const& reporterAndFile : config->getReportersAndOutputFiles()) {
auto& stream = config->getReporterOutputStream(reporterIdx);
multi->addReporter(createReporter(reporterAndFile.reporterName, ReporterConfig(config, stream)));
reporterIdx++;
}
return multi;
}

2
src/catch2/interfaces/catch_interfaces_config.hpp
View File

@ -61,7 +61,7 @@ namespace Catch {
virtual ~IConfig();
virtual bool allowThrows() const = 0;
virtual std::ostream& stream() const = 0;
virtual std::ostream& defaultStream() const = 0;
virtual StringRef name() const = 0;
virtual bool includeSuccessfulResults() const = 0;
virtual bool shouldDebugBreak() const = 0;

3
src/catch2/interfaces/catch_interfaces_reporter.cpp
View File

@ -20,9 +20,6 @@
namespace Catch {
ReporterConfig::ReporterConfig( IConfig const* _fullConfig )
: m_stream( &_fullConfig->stream() ), m_fullConfig( _fullConfig ) {}
ReporterConfig::ReporterConfig( IConfig const* _fullConfig, std::ostream& _stream )
: m_stream( &_stream ), m_fullConfig( _fullConfig ) {}

2
src/catch2/interfaces/catch_interfaces_reporter.hpp
View File

@ -32,8 +32,6 @@ namespace Catch {
struct IConfig;
struct ReporterConfig {
explicit ReporterConfig( IConfig const* _fullConfig );
ReporterConfig( IConfig const* _fullConfig, std::ostream& _stream );
std::ostream& stream() const;

3
src/catch2/interfaces/catch_interfaces_reporter_registry.hpp
View File

@ -23,13 +23,14 @@ namespace Catch {
using IStreamingReporterPtr = Detail::unique_ptr<IStreamingReporter>;
struct IReporterFactory;
using IReporterFactoryPtr = Detail::unique_ptr<IReporterFactory>;
struct ReporterConfig;
struct IReporterRegistry {
using FactoryMap = std::map<std::string, IReporterFactoryPtr, Detail::CaseInsensitiveLess>;
using Listeners = std::vector<IReporterFactoryPtr>;
virtual ~IReporterRegistry(); // = default
virtual IStreamingReporterPtr create( std::string const& name, IConfig const* config ) const = 0;
virtual IStreamingReporterPtr create( std::string const& name, ReporterConfig const& config ) const = 0;
virtual FactoryMap const& getFactories() const = 0;
virtual Listeners const& getListeners() const = 0;
};

60
src/catch2/internal/catch_commandline.cpp
View File

@ -14,6 +14,7 @@
#include <catch2/interfaces/catch_interfaces_reporter_registry.hpp>
#include <catch2/interfaces/catch_interfaces_reporter.hpp>
#include <algorithm>
#include <fstream>
#include <string>
@ -133,15 +134,56 @@ namespace Catch {
return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' );
return ParserResult::ok( ParseResultType::Matched );
};
auto const setReporter = [&]( std::string const& reporter ) {
auto const setReporter = [&]( std::string const& reporterSpec ) {
IReporterRegistry::FactoryMap const& factories = getRegistryHub().getReporterRegistry().getFactories();
auto result = factories.find( reporter );
// clear the default reporter
if (!config._nonDefaultReporterSpecifications) {
config.reporterSpecifications.clear();
config._nonDefaultReporterSpecifications = true;
}
// Exactly one of the reporters may be specified without an output
// file, in which case it defaults to the output specified by "-o"
// (or standard output).
static constexpr auto separator = "::";
static constexpr size_t separatorSize = 2;
auto separatorPos = reporterSpec.find( separator );
const bool containsFileName = separatorPos != reporterSpec.npos;
std::string reporterName;
Optional<std::string> outputFileName;
if (!containsFileName) {
reporterName = reporterSpec;
} else {
reporterName = reporterSpec.substr( 0, separatorPos );
outputFileName = reporterSpec.substr(
separatorPos + separatorSize, reporterSpec.size() );
}
auto result = factories.find( reporterName );
if( result == factories.end() )
return ParserResult::runtimeError( "Unrecognized reporter, '" + reporterName + "'. Check available with --list-reporters" );
if( containsFileName && outputFileName->empty() )
return ParserResult::runtimeError( "Reporter '" + reporterName + "' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." );
config.reporterSpecifications.push_back({ std::move(reporterName), std::move(outputFileName) });
// It would be enough to check this only once at the very end, but there is
// not a place where we could call this check, so do it every time it could fail.
// For valid inputs, this is still called at most once.
if (!containsFileName) {
int n_reporters_without_file = 0;
for (auto const& spec : config.reporterSpecifications) {
if (spec.outputFileName.none()) {
n_reporters_without_file++;
}
}
if (n_reporters_without_file > 1) {
return ParserResult::runtimeError( "Only one reporter may have unspecified output file." );
}
}
if( factories.end() != result )
config.reporterName = reporter;
else
return ParserResult::runtimeError( "Unrecognized reporter, '" + reporter + "'. Check available with --list-reporters" );
return ParserResult::ok( ParseResultType::Matched );
};
auto const setShardCount = [&]( std::string const& shardCount ) {
@ -202,10 +244,10 @@ namespace Catch {
| Opt( config.showInvisibles )
["-i"]["--invisibles"]
( "show invisibles (tabs, newlines)" )
| Opt( config.outputFilename, "filename" )
| Opt( config.defaultOutputFilename, "filename" )
["-o"]["--out"]
( "output filename" )
| Opt( setReporter, "name" )
( "default output filename" )
| Opt( accept_many, setReporter, "name[:output-file]" )
["-r"]["--reporter"]
( "reporter to use (defaults to console)" )
| Opt( config.name, "name" )

2
src/catch2/internal/catch_console_colour.cpp
View File

@ -159,7 +159,7 @@ namespace {
void setColour( const char* _escapeCode ) {
// The escape sequence must be flushed to console, otherwise if
// stdin and stderr are intermixed, we'd get accidentally coloured output.
getCurrentContext().getConfig()->stream()
getCurrentContext().getConfig()->defaultStream()
<< '\033' << _escapeCode << std::flush;
}
};

4
src/catch2/internal/catch_reporter_registry.cpp
View File

@ -36,11 +36,11 @@ namespace Catch {
ReporterRegistry::~ReporterRegistry() = default;
IStreamingReporterPtr ReporterRegistry::create( std::string const& name, IConfig const* config ) const {
IStreamingReporterPtr ReporterRegistry::create( std::string const& name, ReporterConfig const& config ) const {
auto it = m_factories.find( name );
if( it == m_factories.end() )
return nullptr;
return it->second->create( ReporterConfig( config ) );
return it->second->create( config );
}
void ReporterRegistry::registerReporter( std::string const& name, IReporterFactoryPtr factory ) {

2
src/catch2/internal/catch_reporter_registry.hpp
View File

@ -21,7 +21,7 @@ namespace Catch {
ReporterRegistry();
~ReporterRegistry() override; // = default, out of line to allow fwd decl
IStreamingReporterPtr create( std::string const& name, IConfig const* config ) const override;
IStreamingReporterPtr create( std::string const& name, ReporterConfig const& config ) const override;
void registerReporter( std::string const& name, IReporterFactoryPtr factory );
void registerListener( IReporterFactoryPtr factory );

157
src/catch2/reporters/catch_reporter_listening.cpp
View File

@ -6,170 +6,185 @@
// SPDX-License-Identifier: BSL-1.0
#include <catch2/reporters/catch_reporter_listening.hpp>
#include <catch2/catch_config.hpp>
#include <catch2/internal/catch_move_and_forward.hpp>
#include <catch2/internal/catch_stream.hpp>
#include <cassert>
namespace Catch {
void ListeningReporter::addListener( IStreamingReporterPtr&& listener ) {
m_listeners.push_back( CATCH_MOVE( listener ) );
void ListeningReporter::updatePreferences(IStreamingReporter const& reporterish) {
m_preferences.shouldRedirectStdOut |=
reporterish.getPreferences().shouldRedirectStdOut;
m_preferences.shouldReportAllAssertions |=
reporterish.getPreferences().shouldReportAllAssertions;
}
void ListeningReporter::addReporter(IStreamingReporterPtr&& reporter) {
assert(!m_reporter && "Listening reporter can wrap only 1 real reporter");
m_reporter = CATCH_MOVE( reporter );
m_preferences.shouldRedirectStdOut = m_reporter->getPreferences().shouldRedirectStdOut;
void ListeningReporter::addListener( IStreamingReporterPtr&& listener ) {
updatePreferences(*listener);
m_reporterLikes.insert(m_reporterLikes.begin() + m_insertedListeners, CATCH_MOVE(listener) );
++m_insertedListeners;
}
void ListeningReporter::addReporter( IStreamingReporterPtr&& reporter ) {
updatePreferences(*reporter);
// We will need to output the captured stdout if there are reporters
// that do not want it captured.
// We do not consider listeners, because it is generally assumed that
// listeners are output-transparent, even though they can ask for stdout
// capture to do something with it.
m_haveNoncapturingReporters |= !reporter->getPreferences().shouldRedirectStdOut;
// Reporters can always be placed to the back without breaking the
// reporting order
m_reporterLikes.push_back( CATCH_MOVE( reporter ) );
}
void ListeningReporter::noMatchingTestCases( StringRef unmatchedSpec ) {
for ( auto& listener : m_listeners ) {
listener->noMatchingTestCases( unmatchedSpec );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->noMatchingTestCases( unmatchedSpec );
}
m_reporter->noMatchingTestCases( unmatchedSpec );
}
void ListeningReporter::fatalErrorEncountered( StringRef error ) {
for ( auto& listener : m_listeners ) {
listener->fatalErrorEncountered( error );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->fatalErrorEncountered( error );
}
m_reporter->fatalErrorEncountered( error );
}
void ListeningReporter::reportInvalidTestSpec( StringRef arg ) {
for ( auto& listener : m_listeners ) {
listener->reportInvalidTestSpec( arg );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->reportInvalidTestSpec( arg );
}
m_reporter->reportInvalidTestSpec( arg );
}
void ListeningReporter::benchmarkPreparing( StringRef name ) {
for (auto& listener : m_listeners) {
listener->benchmarkPreparing(name);
for (auto& reporterish : m_reporterLikes) {
reporterish->benchmarkPreparing(name);
}
m_reporter->benchmarkPreparing(name);
}
void ListeningReporter::benchmarkStarting( BenchmarkInfo const& benchmarkInfo ) {
for ( auto& listener : m_listeners ) {
listener->benchmarkStarting( benchmarkInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->benchmarkStarting( benchmarkInfo );
}
m_reporter->benchmarkStarting( benchmarkInfo );
}
void ListeningReporter::benchmarkEnded( BenchmarkStats<> const& benchmarkStats ) {
for ( auto& listener : m_listeners ) {
listener->benchmarkEnded( benchmarkStats );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->benchmarkEnded( benchmarkStats );
}
m_reporter->benchmarkEnded( benchmarkStats );
}
void ListeningReporter::benchmarkFailed( StringRef error ) {
for (auto& listener : m_listeners) {
listener->benchmarkFailed(error);
for (auto& reporterish : m_reporterLikes) {
reporterish->benchmarkFailed(error);
}
m_reporter->benchmarkFailed(error);
}
void ListeningReporter::testRunStarting( TestRunInfo const& testRunInfo ) {
for ( auto& listener : m_listeners ) {
listener->testRunStarting( testRunInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testRunStarting( testRunInfo );
}
m_reporter->testRunStarting( testRunInfo );
}
void ListeningReporter::testCaseStarting( TestCaseInfo const& testInfo ) {
for ( auto& listener : m_listeners ) {
listener->testCaseStarting( testInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testCaseStarting( testInfo );
}
m_reporter->testCaseStarting( testInfo );
}
void
ListeningReporter::testCasePartialStarting( TestCaseInfo const& testInfo,
uint64_t partNumber ) {
for ( auto& listener : m_listeners ) {
listener->testCasePartialStarting( testInfo, partNumber );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testCasePartialStarting( testInfo, partNumber );
}
m_reporter->testCasePartialStarting( testInfo, partNumber );
}
void ListeningReporter::sectionStarting( SectionInfo const& sectionInfo ) {
for ( auto& listener : m_listeners ) {
listener->sectionStarting( sectionInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->sectionStarting( sectionInfo );
}
m_reporter->sectionStarting( sectionInfo );
}
void ListeningReporter::assertionStarting( AssertionInfo const& assertionInfo ) {
for ( auto& listener : m_listeners ) {
listener->assertionStarting( assertionInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->assertionStarting( assertionInfo );
}
m_reporter->assertionStarting( assertionInfo );
}
// The return value indicates if the messages buffer should be cleared:
void ListeningReporter::assertionEnded( AssertionStats const& assertionStats ) {
for( auto& listener : m_listeners ) {
listener->assertionEnded( assertionStats );
const bool reportByDefault =
assertionStats.assertionResult.getResultType() != ResultWas::Ok ||
m_config->includeSuccessfulResults();
for ( auto & reporterish : m_reporterLikes ) {
if ( reportByDefault ||
reporterish->getPreferences().shouldReportAllAssertions ) {
reporterish->assertionEnded( assertionStats );
}
}
m_reporter->assertionEnded( assertionStats );
}
void ListeningReporter::sectionEnded( SectionStats const& sectionStats ) {
for ( auto& listener : m_listeners ) {
listener->sectionEnded( sectionStats );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->sectionEnded( sectionStats );
}
m_reporter->sectionEnded( sectionStats );
}
void ListeningReporter::testCasePartialEnded( TestCaseStats const& testInfo,
void ListeningReporter::testCasePartialEnded( TestCaseStats const& testStats,
uint64_t partNumber ) {
for ( auto& listener : m_listeners ) {
listener->testCasePartialEnded( testInfo, partNumber );
// TODO: Fix handling of stderr/stdout?
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testCasePartialEnded( testStats, partNumber );
}
m_reporter->testCasePartialEnded( testInfo, partNumber );
}
void ListeningReporter::testCaseEnded( TestCaseStats const& testCaseStats ) {
for ( auto& listener : m_listeners ) {
listener->testCaseEnded( testCaseStats );
if ( m_preferences.shouldRedirectStdOut && m_haveNoncapturingReporters ) {
if ( !testCaseStats.stdOut.empty() ) {
Catch::cout() << testCaseStats.stdOut << std::flush;
}
if ( !testCaseStats.stdErr.empty() ) {
Catch::cerr() << testCaseStats.stdErr << std::flush;
}
}
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testCaseEnded( testCaseStats );
}
m_reporter->testCaseEnded( testCaseStats );
}
void ListeningReporter::testRunEnded( TestRunStats const& testRunStats ) {
for ( auto& listener : m_listeners ) {
listener->testRunEnded( testRunStats );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->testRunEnded( testRunStats );
}
m_reporter->testRunEnded( testRunStats );
}
void ListeningReporter::skipTest( TestCaseInfo const& testInfo ) {
for ( auto& listener : m_listeners ) {
listener->skipTest( testInfo );
for ( auto& reporterish : m_reporterLikes ) {
reporterish->skipTest( testInfo );
}
m_reporter->skipTest( testInfo );
}
void ListeningReporter::listReporters(std::vector<ReporterDescription> const& descriptions) {
for (auto& listener : m_listeners) {
listener->listReporters(descriptions);
for (auto& reporterish : m_reporterLikes) {
reporterish->listReporters(descriptions);
}
m_reporter->listReporters(descriptions);
}
void ListeningReporter::listTests(std::vector<TestCaseHandle> const& tests) {
for (auto& listener : m_listeners) {
listener->listTests(tests);
for (auto& reporterish : m_reporterLikes) {
reporterish->listTests(tests);
}
m_reporter->listTests(tests);
}
void ListeningReporter::listTags(std::vector<TagInfo> const& tags) {
for (auto& listener : m_listeners) {
listener->listTags(tags);
for (auto& reporterish : m_reporterLikes) {
reporterish->listTags(tags);
}
m_reporter->listTags(tags);
}
} // end namespace Catch

17
src/catch2/reporters/catch_reporter_listening.hpp
View File

@ -13,9 +13,20 @@
namespace Catch {
class ListeningReporter final : public IStreamingReporter {
using Reporters = std::vector<IStreamingReporterPtr>;
Reporters m_listeners;
IStreamingReporterPtr m_reporter = nullptr;
/*
* Stores all added reporters and listeners
*
* All Listeners are stored before all reporters, and individual
* listeners/reporters are stored in order of insertion.
*/
std::vector<IStreamingReporterPtr> m_reporterLikes;
bool m_haveNoncapturingReporters = false;
// Keep track of how many listeners we have already inserted,
// so that we can insert them into the main vector at the right place
size_t m_insertedListeners = 0;
void updatePreferences(IStreamingReporter const& reporterish);
public:
ListeningReporter( IConfig const* config ):

380
tests/SelfTest/Baselines/automake.sw.multi.approved.txt Normal file
View File

@ -0,0 +1,380 @@
:test-result: PASS # A test name that starts with a #
:test-result: PASS #1005: Comparing pointer to int and long (NULL can be either on various systems)
:test-result: PASS #1027: Bitfields can be captured
:test-result: PASS #1147
:test-result: PASS #1175 - Hidden Test
:test-result: PASS #1238
:test-result: PASS #1245
:test-result: PASS #1319: Sections can have description (even if it is not saved
:test-result: PASS #1403
:test-result: FAIL #1455 - INFO and WARN can start with a linebreak
:test-result: FAIL #1514: stderr/stdout is not captured in tests aborted by an exception
:test-result: PASS #1548
:test-result: PASS #1905 -- test spec parser properly clears internal state between compound tests
:test-result: PASS #1912 -- test spec parser handles escaping
:test-result: PASS #1913 - GENERATE inside a for loop should not keep recreating the generator
:test-result: PASS #1913 - GENERATEs can share a line
:test-result: PASS #1938 - GENERATE after a section
:test-result: PASS #1938 - Section followed by flat generate
:test-result: PASS #1938 - flat generate
:test-result: PASS #1938 - mixed sections and generates
:test-result: PASS #1938 - nested generate
:test-result: PASS #1954 - 7 arg template test case sig compiles - 1, 1, 1, 1, 1, 0, 0
:test-result: PASS #1954 - 7 arg template test case sig compiles - 5, 1, 1, 1, 1, 0, 0
:test-result: PASS #1954 - 7 arg template test case sig compiles - 5, 3, 1, 1, 1, 0, 0
:test-result: PASS #2152 - ULP checks between differently signed values were wrong - double
:test-result: PASS #2152 - ULP checks between differently signed values were wrong - float
:test-result: XFAIL #748 - captures with unexpected exceptions
:test-result: PASS #809
:test-result: PASS #833
:test-result: XFAIL #835 -- errno should not be touched by Catch
:test-result: PASS #872
:test-result: PASS #961 -- Dynamically created sections should all be reported
:test-result: FAIL 'Not' checks that should fail
:test-result: PASS 'Not' checks that should succeed
:test-result: PASS (unimplemented) static bools can be evaluated
:test-result: PASS 3x3x3 ints
:test-result: FAIL A METHOD_AS_TEST_CASE based test run that fails
:test-result: PASS A METHOD_AS_TEST_CASE based test run that succeeds
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<float>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<int>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<float>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<int>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<float>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<int>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<float>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<int>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<float, 6>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<int, 2>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<float, 6>
:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<int, 2>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<float,6>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<int,2>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<float,6>
:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<int,2>
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - double
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - float
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - int
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - double
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - int
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 1
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 3
:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 6
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 1
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 3
:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 6
:test-result: FAIL A TEST_CASE_METHOD based test run that fails
:test-result: PASS A TEST_CASE_METHOD based test run that succeeds
:test-result: PASS A Template product test case - Foo<float>
:test-result: PASS A Template product test case - Foo<int>
:test-result: PASS A Template product test case - std::vector<float>
:test-result: PASS A Template product test case - std::vector<int>
:test-result: PASS A Template product test case with array signature - Bar<float, 42>
:test-result: PASS A Template product test case with array signature - Bar<int, 9>
:test-result: PASS A Template product test case with array signature - std::array<float, 42>
:test-result: PASS A Template product test case with array signature - std::array<int, 9>
:test-result: PASS A comparison that uses literals instead of the normal constructor
:test-result: FAIL A couple of nested sections followed by a failure
:test-result: FAIL A failing expression with a non streamable type is still captured
:test-result: PASS Absolute margin
:test-result: FAIL An empty test with no assertions
:test-result: PASS An expression with side-effects should only be evaluated once
:test-result: FAIL An unchecked exception reports the line of the last assertion
:test-result: PASS Anonymous test case 1
:test-result: PASS Approx setters validate their arguments
:test-result: PASS Approx with exactly-representable margin
:test-result: PASS Approximate PI
:test-result: PASS Approximate comparisons with different epsilons
:test-result: PASS Approximate comparisons with floats
:test-result: PASS Approximate comparisons with ints
:test-result: PASS Approximate comparisons with mixed numeric types
:test-result: PASS Arbitrary predicate matcher
:test-result: PASS Assertion macros support bit operators and bool conversions
:test-result: PASS Assertions then sections
:test-result: PASS Basic use of the Contains range matcher
:test-result: PASS Basic use of the Empty range matcher
:test-result: PASS CAPTURE can deal with complex expressions
:test-result: PASS CAPTURE can deal with complex expressions involving commas
:test-result: PASS CAPTURE parses string and character constants
:test-result: PASS Capture and info messages
:test-result: PASS CaseInsensitiveEqualsTo is case insensitive
:test-result: PASS CaseInsensitiveLess is case insensitive
:test-result: PASS Character pretty printing
:test-result: PASS Clara::Arg supports single-arg parse the way Opt does
:test-result: PASS Clara::Opt supports accept-many lambdas
:test-result: PASS Combining MatchAllOfGeneric does not nest
:test-result: PASS Combining MatchAnyOfGeneric does not nest
:test-result: PASS Combining MatchNotOfGeneric does not nest
:test-result: PASS Combining concrete matchers does not use templated matchers
:test-result: PASS Combining only templated matchers
:test-result: PASS Combining templated and concrete matchers
:test-result: PASS Combining templated matchers
:test-result: PASS Commas in various macros are allowed
:test-result: PASS Comparing function pointers
:test-result: PASS Comparison ops
:test-result: PASS Comparison with explicitly convertible types
:test-result: PASS Comparisons between ints where one side is computed
:test-result: PASS Comparisons between unsigned ints and negative signed ints match c++ standard behaviour
:test-result: PASS Comparisons with int literals don't warn when mixing signed/ unsigned
:test-result: PASS Composed generic matchers shortcircuit
:test-result: PASS Composed matchers shortcircuit
:test-result: FAIL Contains string matcher
:test-result: PASS Copy and then generate a range
:test-result: FAIL Custom exceptions can be translated when testing for nothrow
:test-result: FAIL Custom exceptions can be translated when testing for throwing as something else
:test-result: FAIL Custom std-exceptions can be custom translated
:test-result: PASS Default scale is invisible to comparison
:test-result: PASS Directly creating an EnumInfo
:test-result: PASS Empty tag is not allowed
:test-result: FAIL EndsWith string matcher
:test-result: PASS Enums can quickly have stringification enabled using REGISTER_ENUM
:test-result: PASS Enums in namespaces can quickly have stringification enabled using REGISTER_ENUM
:test-result: PASS Epsilon only applies to Approx's value
:test-result: XFAIL Equality checks that should fail
:test-result: PASS Equality checks that should succeed
:test-result: PASS Equals
:test-result: FAIL Equals string matcher
:test-result: PASS Exception as a value (e.g. in REQUIRE_THROWS_MATCHES) can be stringified
:test-result: FAIL Exception matchers that fail
:test-result: PASS Exception matchers that succeed
:test-result: PASS Exception messages can be tested for
:test-result: PASS Exceptions matchers
:test-result: FAIL Expected exceptions that don't throw or unexpected exceptions fail the test
:test-result: FAIL FAIL aborts the test
:test-result: FAIL FAIL does not require an argument
:test-result: FAIL FAIL_CHECK does not abort the test
:test-result: PASS Factorials are computed
:test-result: PASS Floating point matchers: double
:test-result: PASS Floating point matchers: float
:test-result: PASS Generators -- adapters
:test-result: PASS Generators -- simple
:test-result: PASS Generators internals
:test-result: PASS Greater-than inequalities with different epsilons
:test-result: FAIL INFO and WARN do not abort tests
:test-result: FAIL INFO gets logged on failure
:test-result: FAIL INFO gets logged on failure, even if captured before successful assertions
:test-result: FAIL INFO is reset for each loop
:test-result: XFAIL Inequality checks that should fail
:test-result: PASS Inequality checks that should succeed
:test-result: PASS Lambdas in assertions
:test-result: PASS Less-than inequalities with different epsilons
:test-result: PASS ManuallyRegistered
:test-result: PASS Matchers can be (AllOf) composed with the && operator
:test-result: PASS Matchers can be (AnyOf) composed with the || operator
:test-result: PASS Matchers can be composed with both && and ||
:test-result: FAIL Matchers can be composed with both && and || - failing
:test-result: PASS Matchers can be negated (Not) with the ! operator
:test-result: FAIL Matchers can be negated (Not) with the ! operator - failing
:test-result: XFAIL Mayfail test case with nested sections
:test-result: FAIL Mismatching exception messages failing the test
:test-result: PASS Nested generators and captured variables
:test-result: FAIL Nice descriptive name
:test-result: FAIL Non-std exceptions can be translated
:test-result: PASS Objects that evaluated in boolean contexts can be checked
:test-result: PASS Optionally static assertions
:test-result: FAIL Ordering comparison checks that should fail
:test-result: PASS Ordering comparison checks that should succeed
:test-result: PASS Our PCG implementation provides expected results for known seeds
:test-result: FAIL Output from all sections is reported
:test-result: PASS Overloaded comma or address-of operators are not used
:test-result: PASS Parse test names and tags
:test-result: PASS Parsed tags are matched case insensitive
:test-result: PASS Parsing sharding-related cli flags
:test-result: PASS Parsing tags with non-alphabetical characters is pass-through
:test-result: PASS Parsing warnings
:test-result: PASS Pointers can be compared to null
:test-result: PASS Precision of floating point stringification can be set
:test-result: PASS Predicate matcher can accept const char*
:test-result: PASS Process can be configured on command line
:test-result: PASS Product with differing arities - std::tuple<int, double, float>
:test-result: PASS Product with differing arities - std::tuple<int, double>
:test-result: PASS Product with differing arities - std::tuple<int>
:test-result: PASS Random seed generation accepts known methods
:test-result: PASS Random seed generation reports unknown methods
:test-result: PASS Range type with sentinel
:test-result: FAIL Reconstruction should be based on stringification: #914
:test-result: FAIL Regex string matcher
:test-result: PASS Regression test #1
:test-result: PASS Reporter's write listings to provided stream
:test-result: PASS Reproducer for #2309 - a very long description past 80 chars (default console width) with a late colon : blablabla
:test-result: PASS SUCCEED counts as a test pass
:test-result: PASS SUCCEED does not require an argument
:test-result: PASS Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods
:test-result: PASS Scenario: Do that thing with the thing
:test-result: PASS Scenario: This is a really long scenario name to see how the list command deals with wrapping
:test-result: PASS Scenario: Vector resizing affects size and capacity
:test-result: FAIL Sends stuff to stdout and stderr
:test-result: PASS Some simple comparisons between doubles
:test-result: FAIL Standard output from all sections is reported
:test-result: FAIL StartsWith string matcher
:test-result: PASS Static arrays are convertible to string
:test-result: PASS String matchers
:test-result: PASS StringRef
:test-result: PASS StringRef at compilation time
:test-result: PASS Stringifying char arrays with statically known sizes - char
:test-result: PASS Stringifying char arrays with statically known sizes - signed char
:test-result: PASS Stringifying char arrays with statically known sizes - unsigned char
:test-result: PASS Stringifying std::chrono::duration helpers
:test-result: PASS Stringifying std::chrono::duration with weird ratios
:test-result: PASS Stringifying std::chrono::time_point<system_clock>
:test-result: FAIL Tabs and newlines show in output
:test-result: PASS Tag alias can be registered against tag patterns
:test-result: PASS Tags with spaces and non-alphanumerical characters are accepted
:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 0
:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 1
:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 2
:test-result: PASS Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 0
:test-result: PASS Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 1
:test-result: PASS Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 0
:test-result: PASS Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 1
:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 0
:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 1
:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 2
:test-result: PASS TemplateTest: vectors can be sized and resized - float
:test-result: PASS TemplateTest: vectors can be sized and resized - int
:test-result: PASS TemplateTest: vectors can be sized and resized - std::string
:test-result: PASS TemplateTest: vectors can be sized and resized - std::tuple<int,float>
:test-result: PASS TemplateTestSig: vectors can be sized and resized - (std::tuple<int, float>), 6
:test-result: PASS TemplateTestSig: vectors can be sized and resized - float,4
:test-result: PASS TemplateTestSig: vectors can be sized and resized - int,5
:test-result: PASS TemplateTestSig: vectors can be sized and resized - std::string,15
:test-result: PASS Test case with identical tags keeps just one
:test-result: PASS Test case with one argument
:test-result: PASS Test enum bit values
:test-result: PASS Test with special, characters "in name
:test-result: PASS Testing checked-if
:test-result: XFAIL Testing checked-if 2
:test-result: XFAIL Testing checked-if 3
:test-result: FAIL The NO_FAIL macro reports a failure but does not fail the test
:test-result: PASS The default listing implementation write to provided stream
:test-result: FAIL This test 'should' fail but doesn't
:test-result: FAIL Thrown string literals are translated
:test-result: PASS Tracker
:test-result: PASS Trim strings
:test-result: FAIL Unexpected exceptions can be translated
:test-result: PASS Upcasting special member functions
:test-result: PASS Usage of AllMatch range matcher
:test-result: PASS Usage of AnyMatch range matcher
:test-result: PASS Usage of NoneMatch range matcher
:test-result: PASS Usage of the SizeIs range matcher
:test-result: PASS Use a custom approx
:test-result: PASS Variadic macros
:test-result: PASS Vector Approx matcher
:test-result: FAIL Vector Approx matcher -- failing
:test-result: PASS Vector matchers
:test-result: FAIL Vector matchers that fail
:test-result: PASS When checked exceptions are thrown they can be expected or unexpected
:test-result: FAIL When unchecked exceptions are thrown directly they are always failures
:test-result: FAIL When unchecked exceptions are thrown during a CHECK the test should continue
:test-result: FAIL When unchecked exceptions are thrown during a REQUIRE the test should abort fail
:test-result: FAIL When unchecked exceptions are thrown from functions they are always failures
:test-result: FAIL When unchecked exceptions are thrown from sections they are always failures
:test-result: FAIL When unchecked exceptions are thrown, but caught, they do not affect the test
:test-result: PASS X/level/0/a
:test-result: PASS X/level/0/b
:test-result: PASS X/level/1/a
:test-result: PASS X/level/1/b
:test-result: PASS XmlEncode
:test-result: PASS XmlWriter writes boolean attributes as true/false
:test-result: PASS analyse no analysis
:test-result: PASS array<int, N> -> toString
:test-result: PASS benchmark function call
:test-result: PASS boolean member
:test-result: PASS checkedElse
:test-result: FAIL checkedElse, failing
:test-result: PASS checkedIf
:test-result: FAIL checkedIf, failing
:test-result: PASS classify_outliers
:test-result: PASS comparisons between const int variables
:test-result: PASS comparisons between int variables
:test-result: PASS convertToBits
:test-result: PASS empty tags are not allowed
:test-result: PASS erfc_inv
:test-result: PASS estimate_clock_resolution
:test-result: PASS even more nested SECTION tests
:test-result: FAIL first tag
:test-result: FAIL has printf
:test-result: PASS is_unary_function
:test-result: FAIL just failure
:test-result: FAIL just failure after unscoped info
:test-result: FAIL just info
:test-result: FAIL just unscoped info
:test-result: PASS long long
:test-result: FAIL looped SECTION tests
:test-result: FAIL looped tests
:test-result: PASS make_unique reimplementation
:test-result: PASS mean
:test-result: PASS measure
:test-result: FAIL mix info, unscoped info and warning
:test-result: FAIL more nested SECTION tests
:test-result: PASS nested SECTION tests
:test-result: PASS non streamable - with conv. op
:test-result: PASS non-copyable objects
:test-result: PASS normal_cdf
:test-result: PASS normal_quantile
:test-result: PASS not allowed
:test-result: FAIL not prints unscoped info from previous failures
:test-result: PASS null strings
:test-result: PASS null_ptr
:test-result: PASS pair<pair<int,const char *,pair<std::string,int> > -> toString
:test-result: PASS parseEnums
:test-result: PASS pointer to class
:test-result: PASS print unscoped info if passing unscoped info is printed
:test-result: FAIL prints unscoped info on failure
:test-result: FAIL prints unscoped info only for the first assertion
:test-result: PASS random SECTION tests
:test-result: PASS replaceInPlace
:test-result: PASS resolution
:test-result: PASS run_for_at_least, chronometer
:test-result: PASS run_for_at_least, int
:test-result: FAIL second tag
:test-result: FAIL send a single char to INFO
:test-result: FAIL sends information to INFO
:test-result: PASS shortened hide tags are split apart
:test-result: PASS splitString
:test-result: FAIL stacks unscoped info in loops
:test-result: PASS startsWith
:test-result: PASS std::map is convertible string
:test-result: PASS std::pair<int,const std::string> -> toString
:test-result: PASS std::pair<int,std::string> -> toString
:test-result: PASS std::set is convertible string
:test-result: PASS std::vector<std::pair<std::string,int> > -> toString
:test-result: PASS stringify ranges
:test-result: PASS stringify( has_maker )
:test-result: PASS stringify( has_maker_and_operator )
:test-result: PASS stringify( has_neither )
:test-result: PASS stringify( has_operator )
:test-result: PASS stringify( has_template_operator )
:test-result: PASS stringify( vectors<has_maker> )
:test-result: PASS stringify( vectors<has_maker_and_operator> )
:test-result: PASS stringify( vectors<has_operator> )
:test-result: PASS strlen3
:test-result: PASS tables
:test-result: PASS tags with dots in later positions are not parsed as hidden
:test-result: FAIL thrown std::strings are translated
:test-result: PASS toString on const wchar_t const pointer returns the string contents
:test-result: PASS toString on const wchar_t pointer returns the string contents
:test-result: PASS toString on wchar_t const pointer returns the string contents
:test-result: PASS toString on wchar_t returns the string contents
:test-result: PASS toString(enum class w/operator<<)
:test-result: PASS toString(enum class)
:test-result: PASS toString(enum w/operator<<)
:test-result: PASS toString(enum)
:test-result: PASS tuple<>
:test-result: PASS tuple<float,int>
:test-result: PASS tuple<int>
:test-result: PASS tuple<0,int,const char *>
:test-result: PASS tuple<string,string>
:test-result: PASS tuple<tuple<int>,tuple<>,float>
:test-result: PASS uniform samples
:test-result: PASS unique_ptr reimplementation: basic functionality
:test-result: PASS vec<vec<string,alloc>> -> toString
:test-result: PASS vector<bool> -> toString
:test-result: PASS vector<int,allocator> -> toString
:test-result: PASS vector<int> -> toString
:test-result: PASS vector<string> -> toString
:test-result: PASS vectors can be sized and resized
:test-result: PASS warmup
:test-result: PASS weighted_average_quantile
:test-result: PASS xmlentitycheck

47
tests/SelfTest/Baselines/compact.sw.approved.txt
View File

@ -1250,7 +1250,9 @@ CmdLine.tests.cpp:<line number>: passed: config.processName == "test" for: "test
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.reporterName == "console" for: "console" == "console"
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == std::vector<Catch::ConfigData::ReporterAndFile>{ {"console", {}} } for: { { console, <default-output> } }
==
{ { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
@ -1264,15 +1266,40 @@ CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-r", "console"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterName == "console" for: "console" == "console"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-r", "xml"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterName == "xml" for: "xml" == "xml"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--reporter", "junit"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterName == "junit" for: "junit" == "junit"
CmdLine.tests.cpp:<line number>: passed: !(cli.parse({ "test", "-r", "xml", "-r", "junit" })) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } }
==
{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } }
==
{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } }
==
{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } }
==
{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak == true for: true == true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--break"}) for: {?}
@ -1298,9 +1325,9 @@ CmdLine.tests.cpp:<line number>: passed: config.noThrow for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--nothrow"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.noThrow for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-o", "filename.ext"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--out", "filename.ext"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-abe"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.abortAfter == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak for: true

2480
tests/SelfTest/Baselines/compact.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

2
tests/SelfTest/Baselines/console.std.approved.txt
View File

@ -1396,5 +1396,5 @@ due to unexpected exception with message:
===============================================================================
test cases: 380 | 304 passed | 69 failed | 7 failed as expected
assertions: 2183 | 2028 passed | 128 failed | 27 failed as expected
assertions: 2194 | 2039 passed | 128 failed | 27 failed as expected

191
tests/SelfTest/Baselines/console.sw.approved.txt
View File

@ -9158,9 +9158,11 @@ with expansion:
false == false
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( config.reporterName == "console" )
CHECK( config.reporterSpecifications == std::vector<Catch::ConfigData::ReporterAndFile>{ {"console", {}} } )
with expansion:
"console" == "console"
{ { console, <default-output> } }
==
{ { console, <default-output> } }
CmdLine.tests.cpp:<line number>: PASSED:
CHECK_FALSE( cfg.hasTestFilters() )
@ -9260,14 +9262,20 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({"test", "-r", "console"}) )
CHECK( result )
with expansion:
{?}
with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterName == "console" )
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } )
with expansion:
"console" == "console"
{ { console, <default-output> } }
==
{ { console, <default-output> } }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9278,14 +9286,20 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({"test", "-r", "xml"}) )
CHECK( result )
with expansion:
{?}
with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterName == "xml" )
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } )
with expansion:
"xml" == "xml"
{ { xml, <default-output> } }
==
{ { xml, <default-output> } }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9296,27 +9310,20 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({"test", "--reporter", "junit"}) )
CHECK( result )
with expansion:
{?}
with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterName == "junit" )
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } )
with expansion:
"junit" == "junit"
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Only one reporter is accepted
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_FALSE( cli.parse({ "test", "-r", "xml", "-r", "junit" }) )
with expansion:
!{?}
{ { junit, <default-output> } }
==
{ { junit, <default-output> } }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9337,6 +9344,136 @@ with expansion:
"Unrecognized reporter, 'unsupported'. Check available with --list-reporters"
contains: "Unrecognized reporter"
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
With output file
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( result )
with expansion:
{?}
with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } )
with expansion:
{ { console, out.txt } }
==
{ { console, out.txt } }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
With Windows-like absolute path as output file
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( result )
with expansion:
{?}
with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } )
with expansion:
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Output file cannot be empty
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("empty filename") )
with expansion:
"Reporter 'console' has empty filename specified as its output. Supply a
filename or remove the colons to use the default output." contains: "empty
filename"
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Multiple reporters
All with output files
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } )
with expansion:
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Multiple reporters
Mixed output files and default output
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } )
with expansion:
{ { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Multiple reporters
cannot have multiple reporters with default output
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") )
with expansion:
"Only one reporter may have unspecified output file." contains: "Only one
reporter may have unspecified output file."
-------------------------------------------------------------------------------
Process can be configured on command line
debugger
@ -9574,7 +9711,7 @@ with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.outputFilename == "filename.ext" )
REQUIRE( config.defaultOutputFilename == "filename.ext" )
with expansion:
"filename.ext" == "filename.ext"
@ -9592,7 +9729,7 @@ with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.outputFilename == "filename.ext" )
REQUIRE( config.defaultOutputFilename == "filename.ext" )
with expansion:
"filename.ext" == "filename.ext"
@ -17517,5 +17654,5 @@ Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 380 | 290 passed | 83 failed | 7 failed as expected
assertions: 2198 | 2028 passed | 143 failed | 27 failed as expected
assertions: 2209 | 2039 passed | 143 failed | 27 failed as expected

17650
tests/SelfTest/Baselines/console.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

8
tests/SelfTest/Baselines/default.sw.multi.approved.txt Normal file
View File

@ -0,0 +1,8 @@
This would not be caught previously
Nor would this
A string sent directly to stdout
A string sent directly to stderr
A string sent to stderr via clog
Message from section one
Message from section two
loose text artifact

9
tests/SelfTest/Baselines/junit.sw.approved.txt
View File

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact
>
<testsuite name="<exe-name>" errors="17" failures="126" tests="2198" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="126" tests="2209" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@ -1092,8 +1092,13 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/console" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/xml" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/--reporter/junit" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Only one reporter is accepted" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Output file cannot be empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/-b" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/--break" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-a aborts after first failure" time="{duration}" status="run"/>

1902
tests/SelfTest/Baselines/junit.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

7
tests/SelfTest/Baselines/sonarqube.sw.approved.txt
View File

@ -63,8 +63,13 @@
<testCase name="Process can be configured on command line/reporter/-r/console" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/-r/xml" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/--reporter/junit" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Only one reporter is accepted" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Output file cannot be empty" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/debugger/-b" duration="{duration}"/>
<testCase name="Process can be configured on command line/debugger/--break" duration="{duration}"/>
<testCase name="Process can be configured on command line/abort/-a aborts after first failure" duration="{duration}"/>

1920
tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

46
tests/SelfTest/Baselines/tap.sw.approved.txt
View File

@ -2423,7 +2423,7 @@ ok {test-number} - config.abortAfter == -1 for: -1 == -1
# Process can be configured on command line
ok {test-number} - config.noThrow == false for: false == false
# Process can be configured on command line
ok {test-number} - config.reporterName == "console" for: "console" == "console"
ok {test-number} - config.reporterSpecifications == std::vector<Catch::ConfigData::ReporterAndFile>{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } }
# Process can be configured on command line
ok {test-number} - !(cfg.hasTestFilters()) for: !false
# Process can be configured on command line
@ -2451,24 +2451,46 @@ ok {test-number} - cfg.testSpec().matches(*fakeTestCase("test1")) == false for:
# Process can be configured on command line
ok {test-number} - cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "-r", "console"}) for: {?}
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterName == "console" for: "console" == "console"
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "-r", "xml"}) for: {?}
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterName == "xml" for: "xml" == "xml"
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } == { { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "--reporter", "junit"}) for: {?}
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterName == "junit" for: "junit" == "junit"
# Process can be configured on command line
ok {test-number} - !(cli.parse({ "test", "-r", "xml", "-r", "junit" })) for: !{?}
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } == { { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } == { { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } == { { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } } == { { xml, output.xml }, { junit, output-junit.xml } }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } } == { { xml, output.xml }, { console, <default-output> } }
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "-b"}) for: {?}
# Process can be configured on command line
ok {test-number} - config.shouldDebugBreak == true for: true == true
@ -2519,11 +2541,11 @@ ok {test-number} - config.noThrow for: true
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "-o", "filename.ext"}) for: {?}
# Process can be configured on command line
ok {test-number} - config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
ok {test-number} - config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "--out", "filename.ext"}) for: {?}
# Process can be configured on command line
ok {test-number} - config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
ok {test-number} - config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
# Process can be configured on command line
ok {test-number} - cli.parse({"test", "-abe"}) for: {?}
# Process can be configured on command line
@ -4398,5 +4420,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2198
1..2209

4416
tests/SelfTest/Baselines/tap.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

922
tests/SelfTest/Baselines/teamcity.sw.multi.approved.txt Normal file
View File

@ -0,0 +1,922 @@
##teamcity[testSuiteStarted name='<exe-name>']
##teamcity[testStarted name='# A test name that starts with a #']
##teamcity[testFinished name='# A test name that starts with a #' duration="{duration}"]
##teamcity[testStarted name='#1005: Comparing pointer to int and long (NULL can be either on various systems)']
##teamcity[testFinished name='#1005: Comparing pointer to int and long (NULL can be either on various systems)' duration="{duration}"]
##teamcity[testStarted name='#1027: Bitfields can be captured']
##teamcity[testFinished name='#1027: Bitfields can be captured' duration="{duration}"]
##teamcity[testStarted name='#1147']
##teamcity[testFinished name='#1147' duration="{duration}"]
##teamcity[testStarted name='#1175 - Hidden Test']
##teamcity[testFinished name='#1175 - Hidden Test' duration="{duration}"]
##teamcity[testStarted name='#1238']
##teamcity[testFinished name='#1238' duration="{duration}"]
##teamcity[testStarted name='#1245']
##teamcity[testFinished name='#1245' duration="{duration}"]
##teamcity[testStarted name='#1319: Sections can have description (even if it is not saved']
##teamcity[testFinished name='#1319: Sections can have description (even if it is not saved' duration="{duration}"]
##teamcity[testStarted name='#1403']
##teamcity[testFinished name='#1403' duration="{duration}"]
##teamcity[testStarted name='#1455 - INFO and WARN can start with a linebreak']
##teamcity[testFinished name='#1455 - INFO and WARN can start with a linebreak' duration="{duration}"]
##teamcity[testStarted name='#1514: stderr/stdout is not captured in tests aborted by an exception']
Tricky.tests.cpp:<line number>|nexplicit failure with message:|n "1514"']
##teamcity[testStdOut name='#1514: stderr/stdout is not captured in tests aborted by an exception' out='This would not be caught previously|n']
##teamcity[testStdErr name='#1514: stderr/stdout is not captured in tests aborted by an exception' out='Nor would this|n']
##teamcity[testFinished name='#1514: stderr/stdout is not captured in tests aborted by an exception' duration="{duration}"]
##teamcity[testStarted name='#1548']
##teamcity[testFinished name='#1548' duration="{duration}"]
##teamcity[testStarted name='#1905 -- test spec parser properly clears internal state between compound tests']
##teamcity[testFinished name='#1905 -- test spec parser properly clears internal state between compound tests' duration="{duration}"]
##teamcity[testStarted name='#1912 -- test spec parser handles escaping']
##teamcity[testFinished name='#1912 -- test spec parser handles escaping' duration="{duration}"]
##teamcity[testStarted name='#1913 - GENERATE inside a for loop should not keep recreating the generator']
##teamcity[testFinished name='#1913 - GENERATE inside a for loop should not keep recreating the generator' duration="{duration}"]
##teamcity[testStarted name='#1913 - GENERATEs can share a line']
##teamcity[testFinished name='#1913 - GENERATEs can share a line' duration="{duration}"]
##teamcity[testStarted name='#1938 - GENERATE after a section']
##teamcity[testFinished name='#1938 - GENERATE after a section' duration="{duration}"]
##teamcity[testStarted name='#1938 - Section followed by flat generate']
##teamcity[testFinished name='#1938 - Section followed by flat generate' duration="{duration}"]
##teamcity[testStarted name='#1938 - flat generate']
##teamcity[testFinished name='#1938 - flat generate' duration="{duration}"]
##teamcity[testStarted name='#1938 - mixed sections and generates']
##teamcity[testFinished name='#1938 - mixed sections and generates' duration="{duration}"]
##teamcity[testStarted name='#1938 - nested generate']
##teamcity[testFinished name='#1938 - nested generate' duration="{duration}"]
##teamcity[testStarted name='#1954 - 7 arg template test case sig compiles - 1, 1, 1, 1, 1, 0, 0']
##teamcity[testFinished name='#1954 - 7 arg template test case sig compiles - 1, 1, 1, 1, 1, 0, 0' duration="{duration}"]
##teamcity[testStarted name='#1954 - 7 arg template test case sig compiles - 5, 1, 1, 1, 1, 0, 0']
##teamcity[testFinished name='#1954 - 7 arg template test case sig compiles - 5, 1, 1, 1, 1, 0, 0' duration="{duration}"]
##teamcity[testStarted name='#1954 - 7 arg template test case sig compiles - 5, 3, 1, 1, 1, 0, 0']
##teamcity[testFinished name='#1954 - 7 arg template test case sig compiles - 5, 3, 1, 1, 1, 0, 0' duration="{duration}"]
##teamcity[testStarted name='#2152 - ULP checks between differently signed values were wrong - double']
##teamcity[testFinished name='#2152 - ULP checks between differently signed values were wrong - double' duration="{duration}"]
##teamcity[testStarted name='#2152 - ULP checks between differently signed values were wrong - float']
##teamcity[testFinished name='#2152 - ULP checks between differently signed values were wrong - float' duration="{duration}"]
##teamcity[testStarted name='#748 - captures with unexpected exceptions']
Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"- failure ignore as test marked as |'ok to fail|'|n']
Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"|n REQUIRE_NOTHROW( thisThrows() )|nwith expansion:|n thisThrows()|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='#748 - captures with unexpected exceptions' duration="{duration}"]
##teamcity[testStarted name='#809']
##teamcity[testFinished name='#809' duration="{duration}"]
##teamcity[testStarted name='#833']
##teamcity[testFinished name='#833' duration="{duration}"]
##teamcity[testStarted name='#835 -- errno should not be touched by Catch']
Misc.tests.cpp:<line number>|nexpression failed|n CHECK( f() == 0 )|nwith expansion:|n 1 == 0|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='#835 -- errno should not be touched by Catch' duration="{duration}"]
##teamcity[testStarted name='#872']
##teamcity[testFinished name='#872' duration="{duration}"]
##teamcity[testStarted name='#961 -- Dynamically created sections should all be reported']
##teamcity[testFinished name='#961 -- Dynamically created sections should all be reported' duration="{duration}"]
##teamcity[testStarted name='|'Not|' checks that should fail']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( false != false )|nwith expansion:|n false != false|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( true != true )|nwith expansion:|n true != true|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( !true )|nwith expansion:|n false|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK_FALSE( true )|nwith expansion:|n !true|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( !trueValue )|nwith expansion:|n false|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK_FALSE( trueValue )|nwith expansion:|n !true|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( !(1 == 1) )|nwith expansion:|n false|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK_FALSE( 1 == 1 )|nwith expansion:|n !(1 == 1)|n']
##teamcity[testFinished name='|'Not|' checks that should fail' duration="{duration}"]
##teamcity[testStarted name='|'Not|' checks that should succeed']
##teamcity[testFinished name='|'Not|' checks that should succeed' duration="{duration}"]
##teamcity[testStarted name='(unimplemented) static bools can be evaluated']
##teamcity[testFinished name='(unimplemented) static bools can be evaluated' duration="{duration}"]
##teamcity[testStarted name='3x3x3 ints']
##teamcity[testFinished name='3x3x3 ints' duration="{duration}"]
##teamcity[testStarted name='A METHOD_AS_TEST_CASE based test run that fails']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( s == "world" )|nwith expansion:|n "hello" == "world"|n']
##teamcity[testFinished name='A METHOD_AS_TEST_CASE based test run that fails' duration="{duration}"]
##teamcity[testStarted name='A METHOD_AS_TEST_CASE based test run that succeeds']
##teamcity[testFinished name='A METHOD_AS_TEST_CASE based test run that succeeds' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<float>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>::m_a.size() == 1 )|nwith expansion:|n 0 == 1|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<float>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<int>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>::m_a.size() == 1 )|nwith expansion:|n 0 == 1|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<int>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<float>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>::m_a.size() == 1 )|nwith expansion:|n 0 == 1|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<float>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<int>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>::m_a.size() == 1 )|nwith expansion:|n 0 == 1|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<int>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<float>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<float>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<int>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<int>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<float>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<float>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<int>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<int>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<float, 6>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>{}.m_a.size() < 2 )|nwith expansion:|n 6 < 2|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<float, 6>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<int, 2>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>{}.m_a.size() < 2 )|nwith expansion:|n 2 < 2|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<int, 2>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<float, 6>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>{}.m_a.size() < 2 )|nwith expansion:|n 6 < 2|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<float, 6>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<int, 2>']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture_2<TestType>{}.m_a.size() < 2 )|nwith expansion:|n 2 < 2|n']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<int, 2>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<float,6>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<float,6>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<int,2>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<int,2>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<float,6>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<float,6>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<int,2>']
##teamcity[testFinished name='A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<int,2>' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - double']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture<TestType>::m_a == 2 )|nwith expansion:|n 1.0 == 2|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - double' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - float']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture<TestType>::m_a == 2 )|nwith expansion:|n 1.0f == 2|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - float' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - int']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Template_Fixture<TestType>::m_a == 2 )|nwith expansion:|n 1 == 2|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that fails - int' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - double']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - double' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - int']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - int' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 1']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Nttp_Fixture<V>::value == 0 )|nwith expansion:|n 1 == 0|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 1' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 3']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Nttp_Fixture<V>::value == 0 )|nwith expansion:|n 3 == 0|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 3' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 6']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( Nttp_Fixture<V>::value == 0 )|nwith expansion:|n 6 == 0|n']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 6' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 1']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 1' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 3']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 3' duration="{duration}"]
##teamcity[testStarted name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 6']
##teamcity[testFinished name='A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 6' duration="{duration}"]
##teamcity[testStarted name='A TEST_CASE_METHOD based test run that fails']
Class.tests.cpp:<line number>|nexpression failed|n REQUIRE( m_a == 2 )|nwith expansion:|n 1 == 2|n']
##teamcity[testFinished name='A TEST_CASE_METHOD based test run that fails' duration="{duration}"]
##teamcity[testStarted name='A TEST_CASE_METHOD based test run that succeeds']
##teamcity[testFinished name='A TEST_CASE_METHOD based test run that succeeds' duration="{duration}"]
##teamcity[testStarted name='A Template product test case - Foo<float>']
##teamcity[testFinished name='A Template product test case - Foo<float>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case - Foo<int>']
##teamcity[testFinished name='A Template product test case - Foo<int>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case - std::vector<float>']
##teamcity[testFinished name='A Template product test case - std::vector<float>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case - std::vector<int>']
##teamcity[testFinished name='A Template product test case - std::vector<int>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case with array signature - Bar<float, 42>']
##teamcity[testFinished name='A Template product test case with array signature - Bar<float, 42>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case with array signature - Bar<int, 9>']
##teamcity[testFinished name='A Template product test case with array signature - Bar<int, 9>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case with array signature - std::array<float, 42>']
##teamcity[testFinished name='A Template product test case with array signature - std::array<float, 42>' duration="{duration}"]
##teamcity[testStarted name='A Template product test case with array signature - std::array<int, 9>']
##teamcity[testFinished name='A Template product test case with array signature - std::array<int, 9>' duration="{duration}"]
##teamcity[testStarted name='A comparison that uses literals instead of the normal constructor']
##teamcity[testFinished name='A comparison that uses literals instead of the normal constructor' duration="{duration}"]
##teamcity[testStarted name='A couple of nested sections followed by a failure']
Misc.tests.cpp:<line number>|nexplicit failure with message:|n "to infinity and beyond"']
##teamcity[testFinished name='A couple of nested sections followed by a failure' duration="{duration}"]
##teamcity[testStarted name='A failing expression with a non streamable type is still captured']
Tricky.tests.cpp:<line number>|nexpression failed|n CHECK( &o1 == &o2 )|nwith expansion:|n 0x<hex digits> == 0x<hex digits>|n']
Tricky.tests.cpp:<line number>|nexpression failed|n CHECK( o1 == o2 )|nwith expansion:|n {?} == {?}|n']
##teamcity[testFinished name='A failing expression with a non streamable type is still captured' duration="{duration}"]
##teamcity[testStarted name='Absolute margin']
##teamcity[testFinished name='Absolute margin' duration="{duration}"]
##teamcity[testStarted name='An empty test with no assertions']
##teamcity[testFinished name='An empty test with no assertions' duration="{duration}"]
##teamcity[testStarted name='An expression with side-effects should only be evaluated once']
##teamcity[testFinished name='An expression with side-effects should only be evaluated once' duration="{duration}"]
##teamcity[testStarted name='An unchecked exception reports the line of the last assertion']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "unexpected exception"|n {Unknown expression after the reported line}|nwith expansion:|n {Unknown expression after the reported line}|n']
##teamcity[testFinished name='An unchecked exception reports the line of the last assertion' duration="{duration}"]
##teamcity[testStarted name='Anonymous test case 1']
##teamcity[testFinished name='Anonymous test case 1' duration="{duration}"]
##teamcity[testStarted name='Approx setters validate their arguments']
##teamcity[testFinished name='Approx setters validate their arguments' duration="{duration}"]
##teamcity[testStarted name='Approx with exactly-representable margin']
##teamcity[testFinished name='Approx with exactly-representable margin' duration="{duration}"]
##teamcity[testStarted name='Approximate PI']
##teamcity[testFinished name='Approximate PI' duration="{duration}"]
##teamcity[testStarted name='Approximate comparisons with different epsilons']
##teamcity[testFinished name='Approximate comparisons with different epsilons' duration="{duration}"]
##teamcity[testStarted name='Approximate comparisons with floats']
##teamcity[testFinished name='Approximate comparisons with floats' duration="{duration}"]
##teamcity[testStarted name='Approximate comparisons with ints']
##teamcity[testFinished name='Approximate comparisons with ints' duration="{duration}"]
##teamcity[testStarted name='Approximate comparisons with mixed numeric types']
##teamcity[testFinished name='Approximate comparisons with mixed numeric types' duration="{duration}"]
##teamcity[testStarted name='Arbitrary predicate matcher']
##teamcity[testFinished name='Arbitrary predicate matcher' duration="{duration}"]
##teamcity[testStarted name='Assertion macros support bit operators and bool conversions']
##teamcity[testFinished name='Assertion macros support bit operators and bool conversions' duration="{duration}"]
##teamcity[testStarted name='Assertions then sections']
##teamcity[testFinished name='Assertions then sections' duration="{duration}"]
##teamcity[testStarted name='Basic use of the Contains range matcher']
##teamcity[testFinished name='Basic use of the Contains range matcher' duration="{duration}"]
##teamcity[testStarted name='Basic use of the Empty range matcher']
##teamcity[testFinished name='Basic use of the Empty range matcher' duration="{duration}"]
##teamcity[testStarted name='CAPTURE can deal with complex expressions']
##teamcity[testFinished name='CAPTURE can deal with complex expressions' duration="{duration}"]
##teamcity[testStarted name='CAPTURE can deal with complex expressions involving commas']
##teamcity[testFinished name='CAPTURE can deal with complex expressions involving commas' duration="{duration}"]
##teamcity[testStarted name='CAPTURE parses string and character constants']
##teamcity[testFinished name='CAPTURE parses string and character constants' duration="{duration}"]
##teamcity[testStarted name='Capture and info messages']
##teamcity[testFinished name='Capture and info messages' duration="{duration}"]
##teamcity[testStarted name='CaseInsensitiveEqualsTo is case insensitive']
##teamcity[testFinished name='CaseInsensitiveEqualsTo is case insensitive' duration="{duration}"]
##teamcity[testStarted name='CaseInsensitiveLess is case insensitive']
##teamcity[testFinished name='CaseInsensitiveLess is case insensitive' duration="{duration}"]
##teamcity[testStarted name='Character pretty printing']
##teamcity[testFinished name='Character pretty printing' duration="{duration}"]
##teamcity[testStarted name='Clara::Arg supports single-arg parse the way Opt does']
##teamcity[testFinished name='Clara::Arg supports single-arg parse the way Opt does' duration="{duration}"]
##teamcity[testStarted name='Clara::Opt supports accept-many lambdas']
##teamcity[testFinished name='Clara::Opt supports accept-many lambdas' duration="{duration}"]
##teamcity[testStarted name='Combining MatchAllOfGeneric does not nest']
##teamcity[testFinished name='Combining MatchAllOfGeneric does not nest' duration="{duration}"]
##teamcity[testStarted name='Combining MatchAnyOfGeneric does not nest']
##teamcity[testFinished name='Combining MatchAnyOfGeneric does not nest' duration="{duration}"]
##teamcity[testStarted name='Combining MatchNotOfGeneric does not nest']
##teamcity[testFinished name='Combining MatchNotOfGeneric does not nest' duration="{duration}"]
##teamcity[testStarted name='Combining concrete matchers does not use templated matchers']
##teamcity[testFinished name='Combining concrete matchers does not use templated matchers' duration="{duration}"]
##teamcity[testStarted name='Combining only templated matchers']
##teamcity[testFinished name='Combining only templated matchers' duration="{duration}"]
##teamcity[testStarted name='Combining templated and concrete matchers']
##teamcity[testFinished name='Combining templated and concrete matchers' duration="{duration}"]
##teamcity[testStarted name='Combining templated matchers']
##teamcity[testFinished name='Combining templated matchers' duration="{duration}"]
##teamcity[testStarted name='Commas in various macros are allowed']
##teamcity[testFinished name='Commas in various macros are allowed' duration="{duration}"]
##teamcity[testStarted name='Comparing function pointers']
##teamcity[testFinished name='Comparing function pointers' duration="{duration}"]
##teamcity[testStarted name='Comparison ops']
##teamcity[testFinished name='Comparison ops' duration="{duration}"]
##teamcity[testStarted name='Comparison with explicitly convertible types']
##teamcity[testFinished name='Comparison with explicitly convertible types' duration="{duration}"]
##teamcity[testStarted name='Comparisons between ints where one side is computed']
##teamcity[testFinished name='Comparisons between ints where one side is computed' duration="{duration}"]
##teamcity[testStarted name='Comparisons between unsigned ints and negative signed ints match c++ standard behaviour']
##teamcity[testFinished name='Comparisons between unsigned ints and negative signed ints match c++ standard behaviour' duration="{duration}"]
##teamcity[testStarted name='Comparisons with int literals don|'t warn when mixing signed/ unsigned']
##teamcity[testFinished name='Comparisons with int literals don|'t warn when mixing signed/ unsigned' duration="{duration}"]
##teamcity[testStarted name='Composed generic matchers shortcircuit']
##teamcity[testFinished name='Composed generic matchers shortcircuit' duration="{duration}"]
##teamcity[testStarted name='Composed matchers shortcircuit']
##teamcity[testFinished name='Composed matchers shortcircuit' duration="{duration}"]
##teamcity[testStarted name='Contains string matcher']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), ContainsSubstring( "not there", Catch::CaseSensitive::No ) )|nwith expansion:|n "this string contains |'abc|' as a substring" contains: "not there" (case insensitive)|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), ContainsSubstring( "STRING" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" contains: "STRING"|n']
##teamcity[testFinished name='Contains string matcher' duration="{duration}"]
##teamcity[testStarted name='Copy and then generate a range']
##teamcity[testFinished name='Copy and then generate a range' duration="{duration}"]
##teamcity[testStarted name='Custom exceptions can be translated when testing for nothrow']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "custom exception - not std"|n REQUIRE_NOTHROW( throwCustom() )|nwith expansion:|n throwCustom()|n']
##teamcity[testFinished name='Custom exceptions can be translated when testing for nothrow' duration="{duration}"]
##teamcity[testStarted name='Custom exceptions can be translated when testing for throwing as something else']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "custom exception - not std"|n REQUIRE_THROWS_AS( throwCustom(), std::exception )|nwith expansion:|n throwCustom(), std::exception|n']
##teamcity[testFinished name='Custom exceptions can be translated when testing for throwing as something else' duration="{duration}"]
##teamcity[testStarted name='Custom std-exceptions can be custom translated']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "custom std exception"']
##teamcity[testFinished name='Custom std-exceptions can be custom translated' duration="{duration}"]
##teamcity[testStarted name='Default scale is invisible to comparison']
##teamcity[testFinished name='Default scale is invisible to comparison' duration="{duration}"]
##teamcity[testStarted name='Directly creating an EnumInfo']
##teamcity[testFinished name='Directly creating an EnumInfo' duration="{duration}"]
##teamcity[testStarted name='Empty tag is not allowed']
##teamcity[testFinished name='Empty tag is not allowed' duration="{duration}"]
##teamcity[testStarted name='EndsWith string matcher']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), EndsWith( "Substring" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" ends with: "Substring"|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), EndsWith( "this", Catch::CaseSensitive::No ) )|nwith expansion:|n "this string contains |'abc|' as a substring" ends with: "this" (case insensitive)|n']
##teamcity[testFinished name='EndsWith string matcher' duration="{duration}"]
##teamcity[testStarted name='Enums can quickly have stringification enabled using REGISTER_ENUM']
##teamcity[testFinished name='Enums can quickly have stringification enabled using REGISTER_ENUM' duration="{duration}"]
##teamcity[testStarted name='Enums in namespaces can quickly have stringification enabled using REGISTER_ENUM']
##teamcity[testFinished name='Enums in namespaces can quickly have stringification enabled using REGISTER_ENUM' duration="{duration}"]
##teamcity[testStarted name='Epsilon only applies to Approx|'s value']
##teamcity[testFinished name='Epsilon only applies to Approx|'s value' duration="{duration}"]
##teamcity[testStarted name='Equality checks that should fail']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven == 6 )|nwith expansion:|n 7 == 6|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven == 8 )|nwith expansion:|n 7 == 8|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven == 0 )|nwith expansion:|n 7 == 0|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one == Approx( 9.11f ) )|nwith expansion:|n 9.1f == Approx( 9.1099996567 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one == Approx( 9.0f ) )|nwith expansion:|n 9.1f == Approx( 9.0 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one == Approx( 1 ) )|nwith expansion:|n 9.1f == Approx( 1.0 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one == Approx( 0 ) )|nwith expansion:|n 9.1f == Approx( 0.0 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.double_pi == Approx( 3.1415 ) )|nwith expansion:|n 3.1415926535 == Approx( 3.1415 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello == "goodbye" )|nwith expansion:|n "hello" == "goodbye"|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello == "hell" )|nwith expansion:|n "hello" == "hell"|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello == "hello1" )|nwith expansion:|n "hello" == "hello1"|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello.size() == 6 )|nwith expansion:|n 5 == 6|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( x == Approx( 1.301 ) )|nwith expansion:|n 1.3 == Approx( 1.301 )|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='Equality checks that should fail' duration="{duration}"]
##teamcity[testStarted name='Equality checks that should succeed']
##teamcity[testFinished name='Equality checks that should succeed' duration="{duration}"]
##teamcity[testStarted name='Equals']
##teamcity[testFinished name='Equals' duration="{duration}"]
##teamcity[testStarted name='Equals string matcher']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), Equals( "this string contains |'ABC|' as a substring" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" equals: "this string contains |'ABC|' as a substring"|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), Equals( "something else", Catch::CaseSensitive::No ) )|nwith expansion:|n "this string contains |'abc|' as a substring" equals: "something else" (case insensitive)|n']
##teamcity[testFinished name='Equals string matcher' duration="{duration}"]
##teamcity[testStarted name='Exception as a value (e.g. in REQUIRE_THROWS_MATCHES) can be stringified']
##teamcity[testFinished name='Exception as a value (e.g. in REQUIRE_THROWS_MATCHES) can be stringified' duration="{duration}"]
##teamcity[testStarted name='Exception matchers that fail']
Matchers.tests.cpp:<line number>|nno exception was thrown where one was expected|n CHECK_THROWS_MATCHES( doesNotThrow(), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n doesNotThrow(), SpecialException, ExceptionMatcher{ 1 }|n']
Matchers.tests.cpp:<line number>|nno exception was thrown where one was expected|n REQUIRE_THROWS_MATCHES( doesNotThrow(), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n doesNotThrow(), SpecialException, ExceptionMatcher{ 1 }|n']
Matchers.tests.cpp:<line number>|nunexpected exception with message:|n "Unknown exception"|n CHECK_THROWS_MATCHES( throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 }|n']
Matchers.tests.cpp:<line number>|nunexpected exception with message:|n "Unknown exception"|n REQUIRE_THROWS_MATCHES( throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THROWS_MATCHES( throwsSpecialException( 3 ), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n SpecialException::what special exception has value of 1|n']
Matchers.tests.cpp:<line number>|nexpression failed|n REQUIRE_THROWS_MATCHES( throwsSpecialException( 4 ), SpecialException, ExceptionMatcher{ 1 } )|nwith expansion:|n SpecialException::what special exception has value of 1|n']
##teamcity[testFinished name='Exception matchers that fail' duration="{duration}"]
##teamcity[testStarted name='Exception matchers that succeed']
##teamcity[testFinished name='Exception matchers that succeed' duration="{duration}"]
##teamcity[testStarted name='Exception messages can be tested for']
##teamcity[testFinished name='Exception messages can be tested for' duration="{duration}"]
##teamcity[testStarted name='Exceptions matchers']
##teamcity[testFinished name='Exceptions matchers' duration="{duration}"]
##teamcity[testStarted name='Expected exceptions that don|'t throw or unexpected exceptions fail the test']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "expected exception"|n CHECK_THROWS_AS( thisThrows(), std::string )|nwith expansion:|n thisThrows(), std::string|n']
Exception.tests.cpp:<line number>|nno exception was thrown where one was expected|n CHECK_THROWS_AS( thisDoesntThrow(), std::domain_error )|nwith expansion:|n thisDoesntThrow(), std::domain_error|n']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "expected exception"|n CHECK_NOTHROW( thisThrows() )|nwith expansion:|n thisThrows()|n']
##teamcity[testFinished name='Expected exceptions that don|'t throw or unexpected exceptions fail the test' duration="{duration}"]
##teamcity[testStarted name='FAIL aborts the test']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "This is a failure"']
##teamcity[testFinished name='FAIL aborts the test' duration="{duration}"]
##teamcity[testStarted name='FAIL does not require an argument']
Message.tests.cpp:<line number>|nexplicit failure']
##teamcity[testFinished name='FAIL does not require an argument' duration="{duration}"]
##teamcity[testStarted name='FAIL_CHECK does not abort the test']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "This is a failure"']
##teamcity[testFinished name='FAIL_CHECK does not abort the test' duration="{duration}"]
##teamcity[testStarted name='Factorials are computed']
##teamcity[testFinished name='Factorials are computed' duration="{duration}"]
##teamcity[testStarted name='Floating point matchers: double']
##teamcity[testFinished name='Floating point matchers: double' duration="{duration}"]
##teamcity[testStarted name='Floating point matchers: float']
##teamcity[testFinished name='Floating point matchers: float' duration="{duration}"]
##teamcity[testStarted name='Generators -- adapters']
##teamcity[testFinished name='Generators -- adapters' duration="{duration}"]
##teamcity[testStarted name='Generators -- simple']
##teamcity[testFinished name='Generators -- simple' duration="{duration}"]
##teamcity[testStarted name='Generators internals']
##teamcity[testFinished name='Generators internals' duration="{duration}"]
##teamcity[testStarted name='Greater-than inequalities with different epsilons']
##teamcity[testFinished name='Greater-than inequalities with different epsilons' duration="{duration}"]
##teamcity[testStarted name='INFO and WARN do not abort tests']
##teamcity[testFinished name='INFO and WARN do not abort tests' duration="{duration}"]
##teamcity[testStarted name='INFO gets logged on failure']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "this message should be logged"|n "so should this"|n REQUIRE( a == 1 )|nwith expansion:|n 2 == 1|n']
##teamcity[testFinished name='INFO gets logged on failure' duration="{duration}"]
##teamcity[testStarted name='INFO gets logged on failure, even if captured before successful assertions']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "this message may be logged later"|n "this message should be logged"|n CHECK( a == 1 )|nwith expansion:|n 2 == 1|n']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "this message may be logged later"|n "this message should be logged"|n "and this, but later"|n CHECK( a == 0 )|nwith expansion:|n 2 == 0|n']
##teamcity[testFinished name='INFO gets logged on failure, even if captured before successful assertions' duration="{duration}"]
##teamcity[testStarted name='INFO is reset for each loop']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "current counter 10"|n "i := 10"|n REQUIRE( i < 10 )|nwith expansion:|n 10 < 10|n']
##teamcity[testFinished name='INFO is reset for each loop' duration="{duration}"]
##teamcity[testStarted name='Inequality checks that should fail']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven != 7 )|nwith expansion:|n 7 != 7|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one != Approx( 9.1f ) )|nwith expansion:|n 9.1f != Approx( 9.1000003815 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.double_pi != Approx( 3.1415926535 ) )|nwith expansion:|n 3.1415926535 != Approx( 3.1415926535 )|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello != "hello" )|nwith expansion:|n "hello" != "hello"|n- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello.size() != 5 )|nwith expansion:|n 5 != 5|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='Inequality checks that should fail' duration="{duration}"]
##teamcity[testStarted name='Inequality checks that should succeed']
##teamcity[testFinished name='Inequality checks that should succeed' duration="{duration}"]
##teamcity[testStarted name='Lambdas in assertions']
##teamcity[testFinished name='Lambdas in assertions' duration="{duration}"]
##teamcity[testStarted name='Less-than inequalities with different epsilons']
##teamcity[testFinished name='Less-than inequalities with different epsilons' duration="{duration}"]
##teamcity[testStarted name='ManuallyRegistered']
##teamcity[testFinished name='ManuallyRegistered' duration="{duration}"]
##teamcity[testStarted name='Matchers can be (AllOf) composed with the && operator']
##teamcity[testFinished name='Matchers can be (AllOf) composed with the && operator' duration="{duration}"]
##teamcity[testStarted name='Matchers can be (AnyOf) composed with the |||| operator']
##teamcity[testFinished name='Matchers can be (AnyOf) composed with the |||| operator' duration="{duration}"]
##teamcity[testStarted name='Matchers can be composed with both && and ||||']
##teamcity[testFinished name='Matchers can be composed with both && and ||||' duration="{duration}"]
##teamcity[testStarted name='Matchers can be composed with both && and |||| - failing']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), ( ContainsSubstring( "string" ) |||| ContainsSubstring( "different" ) ) && ContainsSubstring( "random" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" ( ( contains: "string" or contains: "different" ) and contains: "random" )|n']
##teamcity[testFinished name='Matchers can be composed with both && and |||| - failing' duration="{duration}"]
##teamcity[testStarted name='Matchers can be negated (Not) with the ! operator']
##teamcity[testFinished name='Matchers can be negated (Not) with the ! operator' duration="{duration}"]
##teamcity[testStarted name='Matchers can be negated (Not) with the ! operator - failing']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), !ContainsSubstring( "substring" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" not contains: "substring"|n']
##teamcity[testFinished name='Matchers can be negated (Not) with the ! operator - failing' duration="{duration}"]
##teamcity[testStarted name='Mayfail test case with nested sections']
Condition.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
Condition.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='Mayfail test case with nested sections' duration="{duration}"]
##teamcity[testStarted name='Mismatching exception messages failing the test']
Exception.tests.cpp:<line number>|nexpression failed|n REQUIRE_THROWS_WITH( thisThrows(), "should fail" )|nwith expansion:|n "expected exception" equals: "should fail"|n']
##teamcity[testFinished name='Mismatching exception messages failing the test' duration="{duration}"]
##teamcity[testStarted name='Nested generators and captured variables']
##teamcity[testFinished name='Nested generators and captured variables' duration="{duration}"]
##teamcity[testStarted name='Nice descriptive name']
##teamcity[testFinished name='Nice descriptive name' duration="{duration}"]
##teamcity[testStarted name='Non-std exceptions can be translated']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "custom exception"']
##teamcity[testFinished name='Non-std exceptions can be translated' duration="{duration}"]
##teamcity[testStarted name='Objects that evaluated in boolean contexts can be checked']
##teamcity[testFinished name='Objects that evaluated in boolean contexts can be checked' duration="{duration}"]
##teamcity[testStarted name='Optionally static assertions']
##teamcity[testFinished name='Optionally static assertions' duration="{duration}"]
##teamcity[testStarted name='Ordering comparison checks that should fail']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven > 7 )|nwith expansion:|n 7 > 7|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven < 7 )|nwith expansion:|n 7 < 7|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven > 8 )|nwith expansion:|n 7 > 8|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven < 6 )|nwith expansion:|n 7 < 6|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven < 0 )|nwith expansion:|n 7 < 0|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven < -1 )|nwith expansion:|n 7 < -1|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven >= 8 )|nwith expansion:|n 7 >= 8|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.int_seven <= 6 )|nwith expansion:|n 7 <= 6|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one < 9 )|nwith expansion:|n 9.1f < 9|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one > 10 )|nwith expansion:|n 9.1f > 10|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.float_nine_point_one > 9.2 )|nwith expansion:|n 9.1f > 9.2|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello > "hello" )|nwith expansion:|n "hello" > "hello"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello < "hello" )|nwith expansion:|n "hello" < "hello"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello > "hellp" )|nwith expansion:|n "hello" > "hellp"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello > "z" )|nwith expansion:|n "hello" > "z"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello < "hellm" )|nwith expansion:|n "hello" < "hellm"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello < "a" )|nwith expansion:|n "hello" < "a"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello >= "z" )|nwith expansion:|n "hello" >= "z"|n']
Condition.tests.cpp:<line number>|nexpression failed|n CHECK( data.str_hello <= "a" )|nwith expansion:|n "hello" <= "a"|n']
##teamcity[testFinished name='Ordering comparison checks that should fail' duration="{duration}"]
##teamcity[testStarted name='Ordering comparison checks that should succeed']
##teamcity[testFinished name='Ordering comparison checks that should succeed' duration="{duration}"]
##teamcity[testStarted name='Our PCG implementation provides expected results for known seeds']
##teamcity[testFinished name='Our PCG implementation provides expected results for known seeds' duration="{duration}"]
##teamcity[testStarted name='Output from all sections is reported']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "Message from section one"']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "Message from section two"']
##teamcity[testFinished name='Output from all sections is reported' duration="{duration}"]
##teamcity[testStarted name='Overloaded comma or address-of operators are not used']
##teamcity[testFinished name='Overloaded comma or address-of operators are not used' duration="{duration}"]
##teamcity[testStarted name='Parse test names and tags']
##teamcity[testFinished name='Parse test names and tags' duration="{duration}"]
##teamcity[testStarted name='Parsed tags are matched case insensitive']
##teamcity[testFinished name='Parsed tags are matched case insensitive' duration="{duration}"]
##teamcity[testStarted name='Parsing sharding-related cli flags']
##teamcity[testFinished name='Parsing sharding-related cli flags' duration="{duration}"]
##teamcity[testStarted name='Parsing tags with non-alphabetical characters is pass-through']
##teamcity[testFinished name='Parsing tags with non-alphabetical characters is pass-through' duration="{duration}"]
##teamcity[testStarted name='Parsing warnings']
##teamcity[testFinished name='Parsing warnings' duration="{duration}"]
##teamcity[testStarted name='Pointers can be compared to null']
##teamcity[testFinished name='Pointers can be compared to null' duration="{duration}"]
##teamcity[testStarted name='Precision of floating point stringification can be set']
##teamcity[testFinished name='Precision of floating point stringification can be set' duration="{duration}"]
##teamcity[testStarted name='Predicate matcher can accept const char*']
##teamcity[testFinished name='Predicate matcher can accept const char*' duration="{duration}"]
##teamcity[testStarted name='Process can be configured on command line']
##teamcity[testFinished name='Process can be configured on command line' duration="{duration}"]
##teamcity[testStarted name='Product with differing arities - std::tuple<int, double, float>']
##teamcity[testFinished name='Product with differing arities - std::tuple<int, double, float>' duration="{duration}"]
##teamcity[testStarted name='Product with differing arities - std::tuple<int, double>']
##teamcity[testFinished name='Product with differing arities - std::tuple<int, double>' duration="{duration}"]
##teamcity[testStarted name='Product with differing arities - std::tuple<int>']
##teamcity[testFinished name='Product with differing arities - std::tuple<int>' duration="{duration}"]
##teamcity[testStarted name='Random seed generation accepts known methods']
##teamcity[testFinished name='Random seed generation accepts known methods' duration="{duration}"]
##teamcity[testStarted name='Random seed generation reports unknown methods']
##teamcity[testFinished name='Random seed generation reports unknown methods' duration="{duration}"]
##teamcity[testStarted name='Range type with sentinel']
##teamcity[testFinished name='Range type with sentinel' duration="{duration}"]
##teamcity[testStarted name='Reconstruction should be based on stringification: #914']
Decomposition.tests.cpp:<line number>|nexpression failed|n CHECK( truthy(false) )|nwith expansion:|n Hey, its truthy!|n']
##teamcity[testFinished name='Reconstruction should be based on stringification: #914' duration="{duration}"]
##teamcity[testStarted name='Regex string matcher']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), Matches( "this STRING contains |'abc|' as a substring" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" matches "this STRING contains |'abc|' as a substring" case sensitively|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), Matches( "contains |'abc|' as a substring" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" matches "contains |'abc|' as a substring" case sensitively|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), Matches( "this string contains |'abc|' as a" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" matches "this string contains |'abc|' as a" case sensitively|n']
##teamcity[testFinished name='Regex string matcher' duration="{duration}"]
##teamcity[testStarted name='Regression test #1']
##teamcity[testFinished name='Regression test #1' duration="{duration}"]
##teamcity[testStarted name='Reporter|'s write listings to provided stream']
##teamcity[testFinished name='Reporter|'s write listings to provided stream' duration="{duration}"]
##teamcity[testStarted name='Reproducer for #2309 - a very long description past 80 chars (default console width) with a late colon : blablabla']
##teamcity[testFinished name='Reproducer for #2309 - a very long description past 80 chars (default console width) with a late colon : blablabla' duration="{duration}"]
##teamcity[testStarted name='SUCCEED counts as a test pass']
##teamcity[testFinished name='SUCCEED counts as a test pass' duration="{duration}"]
##teamcity[testStarted name='SUCCEED does not require an argument']
##teamcity[testFinished name='SUCCEED does not require an argument' duration="{duration}"]
##teamcity[testStarted name='Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods']
##teamcity[testFinished name='Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods' duration="{duration}"]
##teamcity[testStarted name='Scenario: Do that thing with the thing']
##teamcity[testFinished name='Scenario: Do that thing with the thing' duration="{duration}"]
##teamcity[testStarted name='Scenario: This is a really long scenario name to see how the list command deals with wrapping']
##teamcity[testFinished name='Scenario: This is a really long scenario name to see how the list command deals with wrapping' duration="{duration}"]
##teamcity[testStarted name='Scenario: Vector resizing affects size and capacity']
##teamcity[testFinished name='Scenario: Vector resizing affects size and capacity' duration="{duration}"]
##teamcity[testStarted name='Sends stuff to stdout and stderr']
##teamcity[testStdOut name='Sends stuff to stdout and stderr' out='A string sent directly to stdout|n']
##teamcity[testStdErr name='Sends stuff to stdout and stderr' out='A string sent directly to stderr|nA string sent to stderr via clog|n']
##teamcity[testFinished name='Sends stuff to stdout and stderr' duration="{duration}"]
##teamcity[testStarted name='Some simple comparisons between doubles']
##teamcity[testFinished name='Some simple comparisons between doubles' duration="{duration}"]
##teamcity[testStarted name='Standard output from all sections is reported']
##teamcity[testStdOut name='Standard output from all sections is reported' out='Message from section one|nMessage from section two|n']
##teamcity[testFinished name='Standard output from all sections is reported' duration="{duration}"]
##teamcity[testStarted name='StartsWith string matcher']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), StartsWith( "This String" ) )|nwith expansion:|n "this string contains |'abc|' as a substring" starts with: "This String"|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( testStringForMatching(), StartsWith( "string", Catch::CaseSensitive::No ) )|nwith expansion:|n "this string contains |'abc|' as a substring" starts with: "string" (case insensitive)|n']
##teamcity[testFinished name='StartsWith string matcher' duration="{duration}"]
##teamcity[testStarted name='Static arrays are convertible to string']
##teamcity[testFinished name='Static arrays are convertible to string' duration="{duration}"]
##teamcity[testStarted name='String matchers']
##teamcity[testFinished name='String matchers' duration="{duration}"]
##teamcity[testStarted name='StringRef']
##teamcity[testFinished name='StringRef' duration="{duration}"]
##teamcity[testStarted name='StringRef at compilation time']
##teamcity[testFinished name='StringRef at compilation time' duration="{duration}"]
##teamcity[testStarted name='Stringifying char arrays with statically known sizes - char']
##teamcity[testFinished name='Stringifying char arrays with statically known sizes - char' duration="{duration}"]
##teamcity[testStarted name='Stringifying char arrays with statically known sizes - signed char']
##teamcity[testFinished name='Stringifying char arrays with statically known sizes - signed char' duration="{duration}"]
##teamcity[testStarted name='Stringifying char arrays with statically known sizes - unsigned char']
##teamcity[testFinished name='Stringifying char arrays with statically known sizes - unsigned char' duration="{duration}"]
##teamcity[testStarted name='Stringifying std::chrono::duration helpers']
##teamcity[testFinished name='Stringifying std::chrono::duration helpers' duration="{duration}"]
##teamcity[testStarted name='Stringifying std::chrono::duration with weird ratios']
##teamcity[testFinished name='Stringifying std::chrono::duration with weird ratios' duration="{duration}"]
##teamcity[testStarted name='Stringifying std::chrono::time_point<system_clock>']
##teamcity[testFinished name='Stringifying std::chrono::time_point<system_clock>' duration="{duration}"]
##teamcity[testStarted name='Tabs and newlines show in output']
Misc.tests.cpp:<line number>|nexpression failed|n CHECK( s1 == s2 )|nwith expansion:|n "if ($b == 10) {|n $a = 20;|n}"|n==|n"if ($b == 10) {|n $a = 20;|n}|n"|n']
##teamcity[testFinished name='Tabs and newlines show in output' duration="{duration}"]
##teamcity[testStarted name='Tag alias can be registered against tag patterns']
##teamcity[testFinished name='Tag alias can be registered against tag patterns' duration="{duration}"]
##teamcity[testStarted name='Tags with spaces and non-alphanumerical characters are accepted']
##teamcity[testFinished name='Tags with spaces and non-alphanumerical characters are accepted' duration="{duration}"]
##teamcity[testStarted name='Template test case method with test types specified inside std::tuple - MyTypes - 0']
##teamcity[testFinished name='Template test case method with test types specified inside std::tuple - MyTypes - 0' duration="{duration}"]
##teamcity[testStarted name='Template test case method with test types specified inside std::tuple - MyTypes - 1']
##teamcity[testFinished name='Template test case method with test types specified inside std::tuple - MyTypes - 1' duration="{duration}"]
##teamcity[testStarted name='Template test case method with test types specified inside std::tuple - MyTypes - 2']
##teamcity[testFinished name='Template test case method with test types specified inside std::tuple - MyTypes - 2' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 0']
##teamcity[testFinished name='Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 0' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 1']
##teamcity[testFinished name='Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 1' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 0']
##teamcity[testFinished name='Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 0' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 1']
##teamcity[testFinished name='Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 1' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside std::tuple - MyTypes - 0']
##teamcity[testFinished name='Template test case with test types specified inside std::tuple - MyTypes - 0' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside std::tuple - MyTypes - 1']
##teamcity[testFinished name='Template test case with test types specified inside std::tuple - MyTypes - 1' duration="{duration}"]
##teamcity[testStarted name='Template test case with test types specified inside std::tuple - MyTypes - 2']
##teamcity[testFinished name='Template test case with test types specified inside std::tuple - MyTypes - 2' duration="{duration}"]
##teamcity[testStarted name='TemplateTest: vectors can be sized and resized - float']
##teamcity[testFinished name='TemplateTest: vectors can be sized and resized - float' duration="{duration}"]
##teamcity[testStarted name='TemplateTest: vectors can be sized and resized - int']
##teamcity[testFinished name='TemplateTest: vectors can be sized and resized - int' duration="{duration}"]
##teamcity[testStarted name='TemplateTest: vectors can be sized and resized - std::string']
##teamcity[testFinished name='TemplateTest: vectors can be sized and resized - std::string' duration="{duration}"]
##teamcity[testStarted name='TemplateTest: vectors can be sized and resized - std::tuple<int,float>']
##teamcity[testFinished name='TemplateTest: vectors can be sized and resized - std::tuple<int,float>' duration="{duration}"]
##teamcity[testStarted name='TemplateTestSig: vectors can be sized and resized - (std::tuple<int, float>), 6']
##teamcity[testFinished name='TemplateTestSig: vectors can be sized and resized - (std::tuple<int, float>), 6' duration="{duration}"]
##teamcity[testStarted name='TemplateTestSig: vectors can be sized and resized - float,4']
##teamcity[testFinished name='TemplateTestSig: vectors can be sized and resized - float,4' duration="{duration}"]
##teamcity[testStarted name='TemplateTestSig: vectors can be sized and resized - int,5']
##teamcity[testFinished name='TemplateTestSig: vectors can be sized and resized - int,5' duration="{duration}"]
##teamcity[testStarted name='TemplateTestSig: vectors can be sized and resized - std::string,15']
##teamcity[testFinished name='TemplateTestSig: vectors can be sized and resized - std::string,15' duration="{duration}"]
##teamcity[testStarted name='Test case with identical tags keeps just one']
##teamcity[testFinished name='Test case with identical tags keeps just one' duration="{duration}"]
##teamcity[testStarted name='Test case with one argument']
##teamcity[testFinished name='Test case with one argument' duration="{duration}"]
##teamcity[testStarted name='Test enum bit values']
##teamcity[testFinished name='Test enum bit values' duration="{duration}"]
##teamcity[testStarted name='Test with special, characters "in name']
##teamcity[testFinished name='Test with special, characters "in name' duration="{duration}"]
##teamcity[testStarted name='Testing checked-if']
##teamcity[testFinished name='Testing checked-if' duration="{duration}"]
##teamcity[testStarted name='Testing checked-if 2']
Misc.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='Testing checked-if 2' duration="{duration}"]
##teamcity[testStarted name='Testing checked-if 3']
Misc.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='Testing checked-if 3' duration="{duration}"]
##teamcity[testStarted name='The NO_FAIL macro reports a failure but does not fail the test']
##teamcity[testFinished name='The NO_FAIL macro reports a failure but does not fail the test' duration="{duration}"]
##teamcity[testStarted name='The default listing implementation write to provided stream']
##teamcity[testFinished name='The default listing implementation write to provided stream' duration="{duration}"]
##teamcity[testStarted name='This test |'should|' fail but doesn|'t']
##teamcity[testFinished name='This test |'should|' fail but doesn|'t' duration="{duration}"]
##teamcity[testStarted name='Thrown string literals are translated']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "For some reason someone is throwing a string literal!"']
##teamcity[testFinished name='Thrown string literals are translated' duration="{duration}"]
##teamcity[testStarted name='Tracker']
##teamcity[testFinished name='Tracker' duration="{duration}"]
##teamcity[testStarted name='Trim strings']
##teamcity[testFinished name='Trim strings' duration="{duration}"]
##teamcity[testStarted name='Unexpected exceptions can be translated']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "3.14"']
##teamcity[testFinished name='Unexpected exceptions can be translated' duration="{duration}"]
##teamcity[testStarted name='Upcasting special member functions']
##teamcity[testFinished name='Upcasting special member functions' duration="{duration}"]
##teamcity[testStarted name='Usage of AllMatch range matcher']
##teamcity[testFinished name='Usage of AllMatch range matcher' duration="{duration}"]
##teamcity[testStarted name='Usage of AnyMatch range matcher']
##teamcity[testFinished name='Usage of AnyMatch range matcher' duration="{duration}"]
##teamcity[testStarted name='Usage of NoneMatch range matcher']
##teamcity[testFinished name='Usage of NoneMatch range matcher' duration="{duration}"]
##teamcity[testStarted name='Usage of the SizeIs range matcher']
##teamcity[testFinished name='Usage of the SizeIs range matcher' duration="{duration}"]
##teamcity[testStarted name='Use a custom approx']
##teamcity[testFinished name='Use a custom approx' duration="{duration}"]
##teamcity[testStarted name='Variadic macros']
##teamcity[testFinished name='Variadic macros' duration="{duration}"]
##teamcity[testStarted name='Vector Approx matcher']
##teamcity[testFinished name='Vector Approx matcher' duration="{duration}"]
##teamcity[testStarted name='Vector Approx matcher -- failing']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( empty, Approx( t1 ) )|nwith expansion:|n { } is approx: { 1.0, 2.0 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v1, Approx( v2 ) )|nwith expansion:|n { 2.0, 4.0, 6.0 } is approx: { 1.0, 3.0, 5.0 }|n']
##teamcity[testFinished name='Vector Approx matcher -- failing' duration="{duration}"]
##teamcity[testStarted name='Vector matchers']
##teamcity[testFinished name='Vector matchers' duration="{duration}"]
##teamcity[testStarted name='Vector matchers that fail']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v, VectorContains( -1 ) )|nwith expansion:|n { 1, 2, 3 } Contains: -1|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( empty, VectorContains( 1 ) )|nwith expansion:|n { } Contains: 1|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( empty, Contains( v ) )|nwith expansion:|n { } Contains: { 1, 2, 3 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v, Contains( v2 ) )|nwith expansion:|n { 1, 2, 3 } Contains: { 1, 2, 4 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v, Equals( v2 ) )|nwith expansion:|n { 1, 2, 3 } Equals: { 1, 2 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v2, Equals( v ) )|nwith expansion:|n { 1, 2 } Equals: { 1, 2, 3 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( empty, Equals( v ) )|nwith expansion:|n { } Equals: { 1, 2, 3 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v, Equals( empty ) )|nwith expansion:|n { 1, 2, 3 } Equals: { }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( v, UnorderedEquals( empty ) )|nwith expansion:|n { 1, 2, 3 } UnorderedEquals: { }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( empty, UnorderedEquals( v ) )|nwith expansion:|n { } UnorderedEquals: { 1, 2, 3 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( permuted, UnorderedEquals( v ) )|nwith expansion:|n { 1, 3 } UnorderedEquals: { 1, 2, 3 }|n']
Matchers.tests.cpp:<line number>|nexpression failed|n CHECK_THAT( permuted, UnorderedEquals( v ) )|nwith expansion:|n { 3, 1 } UnorderedEquals: { 1, 2, 3 }|n']
##teamcity[testFinished name='Vector matchers that fail' duration="{duration}"]
##teamcity[testStarted name='When checked exceptions are thrown they can be expected or unexpected']
##teamcity[testFinished name='When checked exceptions are thrown they can be expected or unexpected' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown directly they are always failures']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "unexpected exception"']
##teamcity[testFinished name='When unchecked exceptions are thrown directly they are always failures' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown during a CHECK the test should continue']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "expected exception"|n CHECK( thisThrows() == 0 )|nwith expansion:|n thisThrows() == 0|n']
##teamcity[testFinished name='When unchecked exceptions are thrown during a CHECK the test should continue' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown during a REQUIRE the test should abort fail']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "expected exception"|n REQUIRE( thisThrows() == 0 )|nwith expansion:|n thisThrows() == 0|n']
##teamcity[testFinished name='When unchecked exceptions are thrown during a REQUIRE the test should abort fail' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown from functions they are always failures']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "expected exception"|n CHECK( thisThrows() == 0 )|nwith expansion:|n thisThrows() == 0|n']
##teamcity[testFinished name='When unchecked exceptions are thrown from functions they are always failures' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown from sections they are always failures']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "unexpected exception"']
##teamcity[testFinished name='When unchecked exceptions are thrown from sections they are always failures' duration="{duration}"]
##teamcity[testStarted name='When unchecked exceptions are thrown, but caught, they do not affect the test']
##teamcity[testFinished name='When unchecked exceptions are thrown, but caught, they do not affect the test' duration="{duration}"]
##teamcity[testStarted name='X/level/0/a']
##teamcity[testFinished name='X/level/0/a' duration="{duration}"]
##teamcity[testStarted name='X/level/0/b']
##teamcity[testFinished name='X/level/0/b' duration="{duration}"]
##teamcity[testStarted name='X/level/1/a']
##teamcity[testFinished name='X/level/1/a' duration="{duration}"]
##teamcity[testStarted name='X/level/1/b']
##teamcity[testFinished name='X/level/1/b' duration="{duration}"]
##teamcity[testStarted name='XmlEncode']
##teamcity[testFinished name='XmlEncode' duration="{duration}"]
##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false']
##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"]
##teamcity[testStarted name='analyse no analysis']
##teamcity[testFinished name='analyse no analysis' duration="{duration}"]
##teamcity[testStarted name='array<int, N> -> toString']
##teamcity[testFinished name='array<int, N> -> toString' duration="{duration}"]
##teamcity[testStarted name='benchmark function call']
##teamcity[testFinished name='benchmark function call' duration="{duration}"]
##teamcity[testStarted name='boolean member']
##teamcity[testFinished name='boolean member' duration="{duration}"]
##teamcity[testStarted name='checkedElse']
##teamcity[testFinished name='checkedElse' duration="{duration}"]
##teamcity[testStarted name='checkedElse, failing']
Misc.tests.cpp:<line number>|nexpression failed|n REQUIRE( testCheckedElse( false ) )|nwith expansion:|n false|n']
##teamcity[testFinished name='checkedElse, failing' duration="{duration}"]
##teamcity[testStarted name='checkedIf']
##teamcity[testFinished name='checkedIf' duration="{duration}"]
##teamcity[testStarted name='checkedIf, failing']
Misc.tests.cpp:<line number>|nexpression failed|n REQUIRE( testCheckedIf( false ) )|nwith expansion:|n false|n']
##teamcity[testFinished name='checkedIf, failing' duration="{duration}"]
##teamcity[testStarted name='classify_outliers']
##teamcity[testFinished name='classify_outliers' duration="{duration}"]
##teamcity[testStarted name='comparisons between const int variables']
##teamcity[testFinished name='comparisons between const int variables' duration="{duration}"]
##teamcity[testStarted name='comparisons between int variables']
##teamcity[testFinished name='comparisons between int variables' duration="{duration}"]
##teamcity[testStarted name='convertToBits']
##teamcity[testFinished name='convertToBits' duration="{duration}"]
##teamcity[testStarted name='empty tags are not allowed']
##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"]
##teamcity[testStarted name='erfc_inv']
##teamcity[testFinished name='erfc_inv' duration="{duration}"]
##teamcity[testStarted name='estimate_clock_resolution']
##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"]
##teamcity[testStarted name='even more nested SECTION tests']
##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='first tag']
##teamcity[testFinished name='first tag' duration="{duration}"]
##teamcity[testStarted name='has printf']
##teamcity[testFinished name='has printf' duration="{duration}"]
##teamcity[testStarted name='is_unary_function']
##teamcity[testFinished name='is_unary_function' duration="{duration}"]
##teamcity[testStarted name='just failure']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "Previous info should not be seen"']
##teamcity[testFinished name='just failure' duration="{duration}"]
##teamcity[testStarted name='just failure after unscoped info']
Message.tests.cpp:<line number>|nexplicit failure with message:|n "previous unscoped info SHOULD not be seen"']
##teamcity[testFinished name='just failure after unscoped info' duration="{duration}"]
##teamcity[testStarted name='just info']
##teamcity[testFinished name='just info' duration="{duration}"]
##teamcity[testStarted name='just unscoped info']
##teamcity[testFinished name='just unscoped info' duration="{duration}"]
##teamcity[testStarted name='long long']
##teamcity[testFinished name='long long' duration="{duration}"]
##teamcity[testStarted name='looped SECTION tests']
Misc.tests.cpp:<line number>|nexpression failed|n CHECK( b > a )|nwith expansion:|n 0 > 1|n']
Misc.tests.cpp:<line number>|nexpression failed|n CHECK( b > a )|nwith expansion:|n 1 > 1|n']
##teamcity[testFinished name='looped SECTION tests' duration="{duration}"]
##teamcity[testStarted name='looped tests']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[0|] (1) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[1|] (1) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[3|] (3) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[4|] (5) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[6|] (13) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "Testing if fib|[7|] (21) is even"|n CHECK( ( fib|[i|] % 2 ) == 0 )|nwith expansion:|n 1 == 0|n']
##teamcity[testFinished name='looped tests' duration="{duration}"]
##teamcity[testStarted name='make_unique reimplementation']
##teamcity[testFinished name='make_unique reimplementation' duration="{duration}"]
##teamcity[testStarted name='mean']
##teamcity[testFinished name='mean' duration="{duration}"]
##teamcity[testStarted name='measure']
##teamcity[testFinished name='measure' duration="{duration}"]
##teamcity[testStarted name='mix info, unscoped info and warning']
##teamcity[testFinished name='mix info, unscoped info and warning' duration="{duration}"]
##teamcity[testStarted name='more nested SECTION tests']
Misc.tests.cpp:<line number>|nexpression failed|n REQUIRE( a == b )|nwith expansion:|n 1 == 2|n']
##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested SECTION tests']
##teamcity[testFinished name='nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='non streamable - with conv. op']
##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"]
##teamcity[testStarted name='non-copyable objects']
##teamcity[testFinished name='non-copyable objects' duration="{duration}"]
##teamcity[testStarted name='normal_cdf']
##teamcity[testFinished name='normal_cdf' duration="{duration}"]
##teamcity[testStarted name='normal_quantile']
##teamcity[testFinished name='normal_quantile' duration="{duration}"]
##teamcity[testStarted name='not allowed']
##teamcity[testFinished name='not allowed' duration="{duration}"]
##teamcity[testStarted name='not prints unscoped info from previous failures']
Message.tests.cpp:<line number>|nexpression failed with message:|n "this SHOULD be seen"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='not prints unscoped info from previous failures' duration="{duration}"]
##teamcity[testStarted name='null strings']
##teamcity[testFinished name='null strings' duration="{duration}"]
##teamcity[testStarted name='null_ptr']
##teamcity[testFinished name='null_ptr' duration="{duration}"]
##teamcity[testStarted name='pair<pair<int,const char *,pair<std::string,int> > -> toString']
##teamcity[testFinished name='pair<pair<int,const char *,pair<std::string,int> > -> toString' duration="{duration}"]
##teamcity[testStarted name='parseEnums']
##teamcity[testFinished name='parseEnums' duration="{duration}"]
##teamcity[testStarted name='pointer to class']
##teamcity[testFinished name='pointer to class' duration="{duration}"]
##teamcity[testStarted name='print unscoped info if passing unscoped info is printed']
##teamcity[testFinished name='print unscoped info if passing unscoped info is printed' duration="{duration}"]
##teamcity[testStarted name='prints unscoped info on failure']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "this SHOULD be seen"|n "this SHOULD also be seen"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='prints unscoped info on failure' duration="{duration}"]
##teamcity[testStarted name='prints unscoped info only for the first assertion']
Message.tests.cpp:<line number>|nexpression failed with message:|n "this SHOULD be seen only ONCE"|n CHECK( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='prints unscoped info only for the first assertion' duration="{duration}"]
##teamcity[testStarted name='random SECTION tests']
##teamcity[testFinished name='random SECTION tests' duration="{duration}"]
##teamcity[testStarted name='replaceInPlace']
##teamcity[testFinished name='replaceInPlace' duration="{duration}"]
##teamcity[testStarted name='resolution']
##teamcity[testFinished name='resolution' duration="{duration}"]
##teamcity[testStarted name='run_for_at_least, chronometer']
##teamcity[testFinished name='run_for_at_least, chronometer' duration="{duration}"]
##teamcity[testStarted name='run_for_at_least, int']
##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"]
##teamcity[testStarted name='second tag']
##teamcity[testFinished name='second tag' duration="{duration}"]
##teamcity[testStarted name='send a single char to INFO']
Misc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='send a single char to INFO' duration="{duration}"]
##teamcity[testStarted name='sends information to INFO']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "hi"|n "i := 7"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='sends information to INFO' duration="{duration}"]
##teamcity[testStarted name='shortened hide tags are split apart']
##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"]
##teamcity[testStarted name='splitString']
##teamcity[testFinished name='splitString' duration="{duration}"]
##teamcity[testStarted name='stacks unscoped info in loops']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "Count 1 to 3..."|n "1"|n "2"|n "3"|n CHECK( false )|nwith expansion:|n false|n']
Message.tests.cpp:<line number>|nexpression failed with messages:|n "Count 4 to 6..."|n "4"|n "5"|n "6"|n CHECK( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='stacks unscoped info in loops' duration="{duration}"]
##teamcity[testStarted name='startsWith']
##teamcity[testFinished name='startsWith' duration="{duration}"]
##teamcity[testStarted name='std::map is convertible string']
##teamcity[testFinished name='std::map is convertible string' duration="{duration}"]
##teamcity[testStarted name='std::pair<int,const std::string> -> toString']
##teamcity[testFinished name='std::pair<int,const std::string> -> toString' duration="{duration}"]
##teamcity[testStarted name='std::pair<int,std::string> -> toString']
##teamcity[testFinished name='std::pair<int,std::string> -> toString' duration="{duration}"]
##teamcity[testStarted name='std::set is convertible string']
##teamcity[testFinished name='std::set is convertible string' duration="{duration}"]
##teamcity[testStarted name='std::vector<std::pair<std::string,int> > -> toString']
##teamcity[testFinished name='std::vector<std::pair<std::string,int> > -> toString' duration="{duration}"]
##teamcity[testStarted name='stringify ranges']
##teamcity[testFinished name='stringify ranges' duration="{duration}"]
##teamcity[testStarted name='stringify( has_maker )']
##teamcity[testFinished name='stringify( has_maker )' duration="{duration}"]
##teamcity[testStarted name='stringify( has_maker_and_operator )']
##teamcity[testFinished name='stringify( has_maker_and_operator )' duration="{duration}"]
##teamcity[testStarted name='stringify( has_neither )']
##teamcity[testFinished name='stringify( has_neither )' duration="{duration}"]
##teamcity[testStarted name='stringify( has_operator )']
##teamcity[testFinished name='stringify( has_operator )' duration="{duration}"]
##teamcity[testStarted name='stringify( has_template_operator )']
##teamcity[testFinished name='stringify( has_template_operator )' duration="{duration}"]
##teamcity[testStarted name='stringify( vectors<has_maker> )']
##teamcity[testFinished name='stringify( vectors<has_maker> )' duration="{duration}"]
##teamcity[testStarted name='stringify( vectors<has_maker_and_operator> )']
##teamcity[testFinished name='stringify( vectors<has_maker_and_operator> )' duration="{duration}"]
##teamcity[testStarted name='stringify( vectors<has_operator> )']
##teamcity[testFinished name='stringify( vectors<has_operator> )' duration="{duration}"]
##teamcity[testStarted name='strlen3']
##teamcity[testFinished name='strlen3' duration="{duration}"]
##teamcity[testStarted name='tables']
##teamcity[testFinished name='tables' duration="{duration}"]
##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden']
##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"]
##teamcity[testStarted name='thrown std::strings are translated']
Exception.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"']
##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"]
##teamcity[testStarted name='toString on const wchar_t const pointer returns the string contents']
##teamcity[testFinished name='toString on const wchar_t const pointer returns the string contents' duration="{duration}"]
##teamcity[testStarted name='toString on const wchar_t pointer returns the string contents']
##teamcity[testFinished name='toString on const wchar_t pointer returns the string contents' duration="{duration}"]
##teamcity[testStarted name='toString on wchar_t const pointer returns the string contents']
##teamcity[testFinished name='toString on wchar_t const pointer returns the string contents' duration="{duration}"]
##teamcity[testStarted name='toString on wchar_t returns the string contents']
##teamcity[testFinished name='toString on wchar_t returns the string contents' duration="{duration}"]
##teamcity[testStarted name='toString(enum class w/operator<<)']
##teamcity[testFinished name='toString(enum class w/operator<<)' duration="{duration}"]
##teamcity[testStarted name='toString(enum class)']
##teamcity[testFinished name='toString(enum class)' duration="{duration}"]
##teamcity[testStarted name='toString(enum w/operator<<)']
##teamcity[testFinished name='toString(enum w/operator<<)' duration="{duration}"]
##teamcity[testStarted name='toString(enum)']
##teamcity[testFinished name='toString(enum)' duration="{duration}"]
##teamcity[testStarted name='tuple<>']
##teamcity[testFinished name='tuple<>' duration="{duration}"]
##teamcity[testStarted name='tuple<float,int>']
##teamcity[testFinished name='tuple<float,int>' duration="{duration}"]
##teamcity[testStarted name='tuple<int>']
##teamcity[testFinished name='tuple<int>' duration="{duration}"]
##teamcity[testStarted name='tuple<0,int,const char *>']
##teamcity[testFinished name='tuple<0,int,const char *>' duration="{duration}"]
##teamcity[testStarted name='tuple<string,string>']
##teamcity[testFinished name='tuple<string,string>' duration="{duration}"]
##teamcity[testStarted name='tuple<tuple<int>,tuple<>,float>']
##teamcity[testFinished name='tuple<tuple<int>,tuple<>,float>' duration="{duration}"]
##teamcity[testStarted name='uniform samples']
##teamcity[testFinished name='uniform samples' duration="{duration}"]
##teamcity[testStarted name='unique_ptr reimplementation: basic functionality']
##teamcity[testFinished name='unique_ptr reimplementation: basic functionality' duration="{duration}"]
##teamcity[testStarted name='vec<vec<string,alloc>> -> toString']
##teamcity[testFinished name='vec<vec<string,alloc>> -> toString' duration="{duration}"]
##teamcity[testStarted name='vector<bool> -> toString']
##teamcity[testFinished name='vector<bool> -> toString' duration="{duration}"]
##teamcity[testStarted name='vector<int,allocator> -> toString']
##teamcity[testFinished name='vector<int,allocator> -> toString' duration="{duration}"]
##teamcity[testStarted name='vector<int> -> toString']
##teamcity[testFinished name='vector<int> -> toString' duration="{duration}"]
##teamcity[testStarted name='vector<string> -> toString']
##teamcity[testFinished name='vector<string> -> toString' duration="{duration}"]
##teamcity[testStarted name='vectors can be sized and resized']
##teamcity[testFinished name='vectors can be sized and resized' duration="{duration}"]
##teamcity[testStarted name='warmup']
##teamcity[testFinished name='warmup' duration="{duration}"]
##teamcity[testStarted name='weighted_average_quantile']
##teamcity[testFinished name='weighted_average_quantile' duration="{duration}"]
##teamcity[testStarted name='xmlentitycheck']
##teamcity[testFinished name='xmlentitycheck' duration="{duration}"]
##teamcity[testSuiteFinished name='<exe-name>']

229
tests/SelfTest/Baselines/xml.sw.approved.txt
View File

@ -11158,10 +11158,12 @@ Nor would this
</Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterName == "console"
config.reporterSpecifications == std::vector&lt;Catch::ConfigData::ReporterAndFile>{ {"console", {}} }
</Original>
<Expanded>
"console" == "console"
{ { console, &lt;default-output> } }
==
{ { console, &lt;default-output> } }
</Expanded>
</Expression>
<Expression success="true" type="CHECK_FALSE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
@ -11290,20 +11292,28 @@ Nor would this
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="-r/console" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({"test", "-r", "console"})
result
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterName == "console"
config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} }
</Original>
<Expanded>
"console" == "console"
{ { console, &lt;default-output> } }
==
{ { console, &lt;default-output> } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11312,20 +11322,28 @@ Nor would this
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="-r/xml" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({"test", "-r", "xml"})
result
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterName == "xml"
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} }
</Original>
<Expanded>
"xml" == "xml"
{ { xml, &lt;default-output> } }
==
{ { xml, &lt;default-output> } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11334,40 +11352,34 @@ Nor would this
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="--reporter/junit" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({"test", "--reporter", "junit"})
result
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterName == "junit"
config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} }
</Original>
<Expanded>
"junit" == "junit"
{ { junit, &lt;default-output> } }
==
{ { junit, &lt;default-output> } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Only one reporter is accepted" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="REQUIRE_FALSE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!(cli.parse({ "test", "-r", "xml", "-r", "junit" }))
</Original>
<Expanded>
!{?}
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="1" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="must match one of the available ones" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
@ -11390,6 +11402,167 @@ Nor would this
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="With output file" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} }
</Original>
<Expanded>
{ { console, out.txt } }
==
{ { console, out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="With Windows-like absolute path as output file" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Info>
result.errorMessage() := ""
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} }
</Original>
<Expanded>
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Output file cannot be empty" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("empty filename")
</Original>
<Expanded>
"Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Multiple reporters" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" })
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} }
</Original>
<Expanded>
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Multiple reporters" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" })
</Original>
<Expanded>
{?}
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} }
</Original>
<Expanded>
{ { xml, output.xml }, { console, &lt;default-output> } }
==
{ { xml, output.xml }, { console, &lt;default-output> } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Multiple reporters" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="cannot have multiple reporters with default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.")
</Original>
<Expanded>
"Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="debugger" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="-b" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
@ -11681,7 +11854,7 @@ Nor would this
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.outputFilename == "filename.ext"
config.defaultOutputFilename == "filename.ext"
</Original>
<Expanded>
"filename.ext" == "filename.ext"
@ -11703,7 +11876,7 @@ Nor would this
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.outputFilename == "filename.ext"
config.defaultOutputFilename == "filename.ext"
</Original>
<Expanded>
"filename.ext" == "filename.ext"
@ -20561,6 +20734,6 @@ loose text artifact
</Section>
<OverallResult success="true"/>
</TestCase>
<OverallResults successes="2028" failures="143" expectedFailures="27"/>
<OverallResults successes="2039" failures="143" expectedFailures="27"/>
<OverallResultsCases successes="290" failures="83" expectedFailures="7"/>
</Catch2TestRun>

20738
tests/SelfTest/Baselines/xml.sw.multi.approved.txt Normal file
View File

File diff suppressed because it is too large Load Diff

64
tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp
View File

@ -3,12 +3,12 @@
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#include <catch2/catch_config.hpp>
#include <catch2/catch_approx.hpp>
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_string.hpp>
#include <catch2/internal/catch_test_spec_parser.hpp>
#include <catch2/catch_test_case_info.hpp>
#include <catch2/catch_config.hpp>
#include <catch2/internal/catch_commandline.hpp>
#include <catch2/generators/catch_generators.hpp>
@ -345,7 +345,7 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
CHECK(config.shouldDebugBreak == false);
CHECK(config.abortAfter == -1);
CHECK(config.noThrow == false);
CHECK(config.reporterName == "console");
CHECK(config.reporterSpecifications == std::vector<Catch::ConfigData::ReporterAndFile>{ {"console", {}} });
Catch::Config cfg(config);
CHECK_FALSE(cfg.hasTestFilters());
@ -384,23 +384,28 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
}
SECTION("reporter") {
using vec_ReporterAndFile = std::vector<Catch::ConfigData::ReporterAndFile>;
using namespace std::string_literals;
SECTION("-r/console") {
CHECK(cli.parse({"test", "-r", "console"}));
auto result = cli.parse({"test", "-r", "console"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterName == "console");
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} });
}
SECTION("-r/xml") {
CHECK(cli.parse({"test", "-r", "xml"}));
auto result = cli.parse({"test", "-r", "xml"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterName == "xml");
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} });
}
SECTION("--reporter/junit") {
CHECK(cli.parse({"test", "--reporter", "junit"}));
auto result = cli.parse({"test", "--reporter", "junit"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterName == "junit");
}
SECTION("Only one reporter is accepted") {
REQUIRE_FALSE(cli.parse({ "test", "-r", "xml", "-r", "junit" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} });
}
SECTION("must match one of the available ones") {
auto result = cli.parse({"test", "--reporter", "unsupported"});
@ -408,6 +413,39 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Unrecognized reporter"));
}
SECTION("With output file") {
auto result = cli.parse({ "test", "-r", "console::out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} });
}
SECTION("With Windows-like absolute path as output file") {
auto result = cli.parse({ "test", "-r", "console::C:\\Temp\\out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} });
}
SECTION("Output file cannot be empty") {
auto result = cli.parse({"test", "--reporter", "console::"});
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("empty filename"));
}
SECTION("Multiple reporters") {
SECTION("All with output files") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} });
}
SECTION("Mixed output files and default output") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} });
}
SECTION("cannot have multiple reporters with default output") {
auto result = cli.parse({ "test", "-r", "console", "-r", "xml::output.xml", "-r", "junit" });
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file."));
}
}
}
SECTION("debugger") {
@ -483,12 +521,12 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
SECTION("-o filename") {
CHECK(cli.parse({"test", "-o", "filename.ext"}));
REQUIRE(config.outputFilename == "filename.ext");
REQUIRE(config.defaultOutputFilename == "filename.ext");
}
SECTION("--out") {
CHECK(cli.parse({"test", "--out", "filename.ext"}));
REQUIRE(config.outputFilename == "filename.ext");
REQUIRE(config.defaultOutputFilename == "filename.ext");
}
}

45
tools/scripts/approvalTests.py
View File

@ -164,19 +164,36 @@ def filterLine(line, isCompact):
return line
def approve(baseName, args):
global overallResult
def get_rawResultsPath(baseName):
return os.path.join(rootPath, '_{0}.tmp'.format(baseName))
def get_baselinesPath(baseName):
return os.path.join(rootPath, '{0}.approved.txt'.format(baseName))
def get_filteredResultsPath(baseName):
return os.path.join(rootPath, '{0}.unapproved.txt'.format(baseName))
def run_test(baseName, args):
args[0:0] = [cmdPath]
if not os.path.exists(cmdPath):
raise Exception("Executable doesn't exist at " + cmdPath)
baselinesPath = os.path.join(rootPath, '{0}.approved.txt'.format(baseName))
rawResultsPath = os.path.join(rootPath, '_{0}.tmp'.format(baseName))
filteredResultsPath = os.path.join(rootPath, '{0}.unapproved.txt'.format(baseName))
print(args)
rawResultsPath = get_rawResultsPath(baseName)
f = open(rawResultsPath, 'w')
subprocess.call(args, stdout=f, stderr=f)
f.close()
def check_outputs(baseName):
global overallResult
rawResultsPath = get_rawResultsPath(baseName)
baselinesPath = get_baselinesPath(baseName)
filteredResultsPath = get_filteredResultsPath(baseName)
rawFile = io.open(rawResultsPath, 'r', encoding='utf-8', errors='surrogateescape')
filteredFile = io.open(filteredResultsPath, 'w', encoding='utf-8', errors='surrogateescape')
for line in rawFile:
@ -204,6 +221,11 @@ def approve(baseName, args):
overallResult = 1
def approve(baseName, args):
run_test(baseName, args)
check_outputs(baseName)
print("Running approvals against executable:")
print(" " + cmdPath)
@ -222,6 +244,19 @@ for reporter in reporters:
reporter_args = ['-r', reporter]
approve(filename, common_args + reporter_args)
## All reporters at the same time
common_args = ["~[!nonportable]~[!benchmark]~[approvals] *", "-s", "-w", "NoAssertions", "--order", "lex", "--rng-seed", "1"]
filenames = ['{}.sw.multi'.format(reporter) for reporter in reporters]
reporter_args = []
for reporter, filename in zip(reporters, filenames):
reporter_args += ['-r', '{}::{}'.format(reporter, get_rawResultsPath(filename))]
run_test("default.sw.multi", common_args + reporter_args)
check_outputs("default.sw.multi")
for reporter, filename in zip(reporters, filenames):
check_outputs(filename)
if overallResult != 0:
print("If these differences are expected, run approve.py to approve new baselines.")