3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-25 06:46:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

Integrate the new reporter specs into Catch2

Browse Source 
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.

Closes #339 as it allows per-reporter forcing of ansi colour codes.
This commit is contained in:
Martin Hořeňovský 2022-04-04 21:07:04 +02:00
parent 3c06bcb374
commit 423e1d2ebb
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
29 changed files with 395 additions and 448 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
docs/command-line.md
View File

@ -123,22 +123,33 @@ Test names containing special characters, such as `,` or `[` can specify them on
<a id="choosing-a-reporter-to-use"></a> <a id="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use ## Choosing a reporter to use
<pre>-r, --reporter &lt;reporter[::output-file]&gt;</pre> <pre>-r, --reporter &lt;reporter[::key=value]*&gt;</pre>
> Support for providing output-file through the `-r`, `--reporter` flag was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
Reporters are how the output from Catch2 (results of assertions, tests, Reporters are how the output from Catch2 (results of assertions, tests,
benchmarks and so on) is formatted and written out. The default reporter benchmarks and so on) is formatted and written out. The default reporter
is called the "Console" reporter and is intended to provide relatively is called the "Console" reporter and is intended to provide relatively
verbose and human-friendly output. verbose and human-friendly output.
Reporters are also individually configurable. To pass configuration options
to the reporter, you append `::key=value` to the reporter specification
as many times as you want, e.g. `--reporter xml::out=someFile.xml`.
The keys must either be prefixed by "X", in which case they are not parsed
by Catch2 and are only passed down to the reporter, or one of options
hardcoded into Catch2. Currently there are only 2,
["out"](#sending-output-to-a-file), and ["colour-mode"](#colour-mode).
_Note that the reporter might still check the X-prefixed options for
validity, and throw an error if they are wrong._
> Support for passing arguments to reporters through the `-r`, `--reporter` flag was introduced in Catch2 X.Y.Z
There are multiple built-in reporters, you can see what they do by using the There are multiple built-in reporters, you can see what they do by using the
[`--list-reporter`](command-line.md#listing-available-tests-tags-or-reporters) [`--list-reporter`](command-line.md#listing-available-tests-tags-or-reporters)
flag. If you need a reporter providing custom format outside of the already flag. If you need a reporter providing custom format outside of the already
provided ones, look at the ["write your own reporter" part of the reporter provided ones, look at the ["write your own reporter" part of the reporter
documentation](reporters.md#writing-your-own-reporter). documentation](reporters.md#writing-your-own-reporter).
This option may be passed multiple times to use multiple (different) This option may be passed multiple times to use multiple (different)
reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters) reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters)
for details on what the resulting behaviour is. Also note that at most one for details on what the resulting behaviour is. Also note that at most one
@ -148,13 +159,12 @@ the [`-o`, `--out`](#sending-output-to-a-file) option.
> Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z > Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
As with the `--out` option, using `-` for the output file name sends the
output to stdout.
_Note: There is currently no way to escape `::` in the reporter spec, _Note: There is currently no way to escape `::` in the reporter spec,
and thus reporter/file names with `::` in them will not work properly. and thus the reporter names, or configuration keys and values, cannot
As `::` in paths is relatively obscure (unlike `:`), we do not consider contain `::`. As `::` in paths is relatively obscure (unlike ':'), we do
this an issue._ not consider this an issue._
<a id="breaking-into-the-debugger"></a> <a id="breaking-into-the-debugger"></a>
## Breaking into the debugger ## Breaking into the debugger

27
docs/reporters.md
View File

@ -43,9 +43,14 @@ them write into different destinations. The two main uses of this are
Specifying multiple reporter looks like this: Specifying multiple reporter looks like this:
``` ```
--reporter console::- --reporter JUnit::result-junit.xml --reporter JUnit::out=result-junit.xml --reporter console::out=-::colour-mode=ansi
``` ```
This tells Catch2 to use two reporters, `JUnit` reporter that writes
its machine-readable XML output to file `result-junit.xml`, and the
`console` reporter that writes its user-friendly output to stdout and
uses ANSI colour codes for colouring the output.
Using multiple reporters (or one reporter and one-or-more [event Using multiple reporters (or one reporter and one-or-more [event
listeners](event-listener.md#top)) can have surprisingly complex semantics listeners](event-listener.md#top)) can have surprisingly complex semantics
when using customization points provided to reporters by Catch2, namely when using customization points provided to reporters by Catch2, namely
@ -162,6 +167,26 @@ Currently there are two customization options:
format includes passing assertions even without the `-s` flag. format includes passing assertions even without the `-s` flag.
### Per-reporter configuration
> Per-reporter configuration was introduced in Catch2 X.Y.Z
Catch2 supports some configuration to happen per reporter. The configuration
options fall into one of two categories:
* Catch2-recognized options
* Reporter-specific options
The former is a small set of universal options that Catch2 handles for
the reporters, e.g. output file or console colour mode. The latter are
options that the reporters have to handle themselves, but the keys and
values can be arbitrary strings, as long as they don't contain `::`. This
allows writing reporters that can be significantly customized at runtime.
Reporter-specific options always have to be prefixed with "X" (large
letter X).
### Other expected functionality of a reporter ### Other expected functionality of a reporter
When writing a custom reporter, there are few more things that you should When writing a custom reporter, there are few more things that you should

21
src/catch2/catch_config.cpp
View File

@ -33,17 +33,6 @@ namespace Catch {
} // unnamed namespace } // unnamed namespace
} // namespace Detail } // namespace Detail
std::ostream& operator<<( std::ostream& os,
ConfigData::ReporterAndFile const& reporter ) {
os << "{ " << reporter.reporterName << ", ";
if ( reporter.outputFileName ) {
os << *reporter.outputFileName;
} else {
os << "<default-output>";
}
return os << " }";
}
Config::Config( ConfigData const& data ): Config::Config( ConfigData const& data ):
m_data( data ) { m_data( data ) {
// We need to trim filter specs to avoid trouble with superfluous // We need to trim filter specs to avoid trouble with superfluous
@ -76,14 +65,14 @@ namespace Catch {
#else #else
"console", "console",
#endif #endif
{} {}, {}, {}
} ); } );
} }
bool defaultOutputUsed = false; bool defaultOutputUsed = false;
m_reporterStreams.reserve( m_data.reporterSpecifications.size() ); m_reporterStreams.reserve( m_data.reporterSpecifications.size() );
for ( auto const& reporterAndFile : m_data.reporterSpecifications ) { for ( auto const& reporterSpec : m_data.reporterSpecifications ) {
if ( reporterAndFile.outputFileName.none() ) { if ( reporterSpec.outputFile().none() ) {
CATCH_ENFORCE( !defaultOutputUsed, CATCH_ENFORCE( !defaultOutputUsed,
"Internal error: cannot use default output for " "Internal error: cannot use default output for "
"multiple reporters" ); "multiple reporters" );
@ -93,7 +82,7 @@ namespace Catch {
openStream( data.defaultOutputFilename ) ); openStream( data.defaultOutputFilename ) );
} else { } else {
m_reporterStreams.push_back( m_reporterStreams.push_back(
openStream( *reporterAndFile.outputFileName ) ); openStream( *reporterSpec.outputFile() ) );
} }
} }
} }
@ -108,7 +97,7 @@ namespace Catch {
std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; } std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; }
std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; } std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; }
std::vector<ConfigData::ReporterAndFile> const& Config::getReportersAndOutputFiles() const { std::vector<ReporterSpec> const& Config::getReporterSpecs() const {
return m_data.reporterSpecifications; return m_data.reporterSpecifications;
} }

16
src/catch2/catch_config.hpp
View File

@ -14,6 +14,7 @@
#include <catch2/internal/catch_unique_ptr.hpp> #include <catch2/internal/catch_unique_ptr.hpp>
#include <catch2/internal/catch_optional.hpp> #include <catch2/internal/catch_optional.hpp>
#include <catch2/internal/catch_random_seed_generation.hpp> #include <catch2/internal/catch_random_seed_generation.hpp>
#include <catch2/internal/catch_reporter_spec_parser.hpp>
#include <iosfwd> #include <iosfwd>
#include <vector> #include <vector>
@ -24,17 +25,6 @@ namespace Catch {
class IStream; class IStream;
struct ConfigData { struct ConfigData {
struct ReporterAndFile {
std::string reporterName;
// If none, the output goes to the default output.
Optional<std::string> outputFileName;
friend bool operator==(ReporterAndFile const& lhs, ReporterAndFile const& rhs) {
return lhs.reporterName == rhs.reporterName && lhs.outputFileName == rhs.outputFileName;
}
friend std::ostream& operator<<(std::ostream &os, ReporterAndFile const& reporter);
};
bool listTests = false; bool listTests = false;
bool listTags = false; bool listTags = false;
@ -72,7 +62,7 @@ namespace Catch {
std::string defaultOutputFilename; std::string defaultOutputFilename;
std::string name; std::string name;
std::string processName; std::string processName;
std::vector<ReporterAndFile> reporterSpecifications; std::vector<ReporterSpec> reporterSpecifications;
std::vector<std::string> testsOrTags; std::vector<std::string> testsOrTags;
std::vector<std::string> sectionsToRun; std::vector<std::string> sectionsToRun;
@ -90,7 +80,7 @@ namespace Catch {
bool listTags() const; bool listTags() const;
bool listReporters() const; bool listReporters() const;
std::vector<ConfigData::ReporterAndFile> const& getReportersAndOutputFiles() const; std::vector<ReporterSpec> const& getReporterSpecs() const;
IStream const* getReporterOutputStream(std::size_t reporterIdx) const; IStream const* getReporterOutputStream(std::size_t reporterIdx) const;
std::vector<std::string> const& getTestsOrTags() const override; std::vector<std::string> const& getTestsOrTags() const override;

25
src/catch2/catch_session.cpp
View File

@ -41,11 +41,18 @@ namespace Catch {
return reporter; return reporter;
} }
IStreamingReporterPtr makeReporter(Config const* config) { IStreamingReporterPtr prepareReporters(Config const* config) {
if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty() if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty()
&& config->getReportersAndOutputFiles().size() == 1) { && config->getReporterSpecs().size() == 1) {
auto const& spec = config->getReporterSpecs()[0];
auto stream = config->getReporterOutputStream(0); auto stream = config->getReporterOutputStream(0);
return createReporter(config->getReportersAndOutputFiles()[0].reporterName, ReporterConfig(config, stream)); return createReporter(
config->getReporterSpecs()[0].name(),
ReporterConfig(
config,
stream,
spec.colourMode().valueOr( config->colourMode() ),
spec.customOptions() ) );
} }
auto multi = Detail::make_unique<MultiReporter>(config); auto multi = Detail::make_unique<MultiReporter>(config);
@ -56,9 +63,15 @@ namespace Catch {
} }
std::size_t reporterIdx = 0; std::size_t reporterIdx = 0;
for (auto const& reporterAndFile : config->getReportersAndOutputFiles()) { for (auto const& reporterSpec : config->getReporterSpecs()) {
auto stream = config->getReporterOutputStream(reporterIdx); auto stream = config->getReporterOutputStream(reporterIdx);
multi->addReporter(createReporter(reporterAndFile.reporterName, ReporterConfig(config, stream))); multi->addReporter( createReporter(
reporterSpec.name(),
ReporterConfig( config,
stream,
reporterSpec.colourMode().valueOr(
config->colourMode() ),
reporterSpec.customOptions() ) ) );
reporterIdx++; reporterIdx++;
} }
@ -304,7 +317,7 @@ namespace Catch {
getCurrentMutableContext().setConfig(m_config.get()); getCurrentMutableContext().setConfig(m_config.get());
// Create reporter(s) so we can route listings through them // Create reporter(s) so we can route listings through them
auto reporter = makeReporter(m_config.get()); auto reporter = prepareReporters(m_config.get());
auto const& invalidSpecs = m_config->testSpec().getInvalidSpecs(); auto const& invalidSpecs = m_config->testSpec().getInvalidSpecs();
if ( !invalidSpecs.empty() ) { if ( !invalidSpecs.empty() ) {

18
src/catch2/interfaces/catch_interfaces_reporter.cpp
View File

@ -14,17 +14,31 @@
#include <catch2/internal/catch_string_manip.hpp> #include <catch2/internal/catch_string_manip.hpp>
#include <catch2/catch_test_case_info.hpp> #include <catch2/catch_test_case_info.hpp>
#include <catch2/reporters/catch_reporter_helpers.hpp> #include <catch2/reporters/catch_reporter_helpers.hpp>
#include <catch2/internal/catch_move_and_forward.hpp>
#include <algorithm> #include <algorithm>
#include <iomanip> #include <iomanip>
namespace Catch { namespace Catch {
ReporterConfig::ReporterConfig( IConfig const* _fullConfig, IStream const* _stream ) ReporterConfig::ReporterConfig(
: m_stream( _stream ), m_fullConfig( _fullConfig ) {} IConfig const* _fullConfig,
IStream const* _stream,
ColourMode colourMode,
std::map<std::string, std::string> customOptions ):
m_stream( _stream ),
m_fullConfig( _fullConfig ),
m_colourMode( colourMode ),
m_customOptions( CATCH_MOVE( customOptions ) ) {}
IStream const* ReporterConfig::stream() const { return m_stream; } IStream const* ReporterConfig::stream() const { return m_stream; }
IConfig const * ReporterConfig::fullConfig() const { return m_fullConfig; } IConfig const * ReporterConfig::fullConfig() const { return m_fullConfig; }
ColourMode ReporterConfig::colourMode() const { return m_colourMode; }
std::map<std::string, std::string> const&
ReporterConfig::customOptions() const {
return m_customOptions;
}
AssertionStats::AssertionStats( AssertionResult const& _assertionResult, AssertionStats::AssertionStats( AssertionResult const& _assertionResult,
std::vector<MessageInfo> const& _infoMessages, std::vector<MessageInfo> const& _infoMessages,

11
src/catch2/interfaces/catch_interfaces_reporter.hpp
View File

@ -19,6 +19,7 @@
#include <catch2/benchmark/catch_outlier_classification.hpp> #include <catch2/benchmark/catch_outlier_classification.hpp>
#include <map>
#include <string> #include <string>
#include <vector> #include <vector>
#include <iosfwd> #include <iosfwd>
@ -31,16 +32,24 @@ namespace Catch {
class TestCaseHandle; class TestCaseHandle;
struct IConfig; struct IConfig;
class IStream; class IStream;
enum class ColourMode : std::uint8_t;
struct ReporterConfig { struct ReporterConfig {
ReporterConfig( IConfig const* _fullConfig, IStream const* _stream ); ReporterConfig( IConfig const* _fullConfig,
IStream const* _stream,
ColourMode colourMode,
std::map<std::string, std::string> customOptions );
IStream const* stream() const; IStream const* stream() const;
IConfig const* fullConfig() const; IConfig const* fullConfig() const;
ColourMode colourMode() const;
std::map<std::string, std::string> const& customOptions() const;
private: private:
IStream const* m_stream; IStream const* m_stream;
IConfig const* m_fullConfig; IConfig const* m_fullConfig;
ColourMode m_colourMode;
std::map<std::string, std::string> m_customOptions;
}; };
struct TestRunInfo { struct TestRunInfo {

61
src/catch2/internal/catch_commandline.cpp
View File

@ -141,57 +141,42 @@ namespace Catch {
return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' ); return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' );
return ParserResult::ok( ParseResultType::Matched ); return ParserResult::ok( ParseResultType::Matched );
}; };
auto const setReporter = [&]( std::string const& reporterSpec ) { auto const setReporter = [&]( std::string const& userReporterSpec ) {
if ( reporterSpec.empty() ) { if ( userReporterSpec.empty() ) {
return ParserResult::runtimeError( "Received empty reporter spec." ); return ParserResult::runtimeError( "Received empty reporter spec." );
} }
// Exactly one of the reporters may be specified without an output Optional<ReporterSpec> parsed =
// file, in which case it defaults to the output specified by "-o" parseReporterSpec( userReporterSpec );
// (or standard output). if ( !parsed ) {
static constexpr auto separator = "::"; return ParserResult::runtimeError(
static constexpr size_t separatorSize = 2; "Could not parse reporter spec '" + userReporterSpec +
auto fileNameSeparatorPos = reporterSpec.find( separator ); "'" );
const bool containsFileName = fileNameSeparatorPos != reporterSpec.npos;
if ( containsFileName ) {
auto nextSeparatorPos = reporterSpec.find(
separator, fileNameSeparatorPos + separatorSize );
if ( nextSeparatorPos != reporterSpec.npos ) {
return ParserResult::runtimeError(
"Too many separators in reporter spec '" + reporterSpec + '\'' );
}
} }
std::string reporterName; auto const& reporterSpec = *parsed;
Optional<std::string> outputFileName;
reporterName = reporterSpec.substr( 0, fileNameSeparatorPos );
if ( reporterName.empty() ) {
return ParserResult::runtimeError( "Reporter name cannot be empty." );
}
if ( containsFileName ) {
outputFileName = reporterSpec.substr(
fileNameSeparatorPos + separatorSize, reporterSpec.size() );
}
IReporterRegistry::FactoryMap const& factories = IReporterRegistry::FactoryMap const& factories =
getRegistryHub().getReporterRegistry().getFactories(); getRegistryHub().getReporterRegistry().getFactories();
auto result = factories.find( reporterName ); auto result = factories.find( reporterSpec.name() );
if( result == factories.end() ) if ( result == factories.end() ) {
return ParserResult::runtimeError( "Unrecognized reporter, '" + reporterName + "'. Check available with --list-reporters" ); return ParserResult::runtimeError(
if( containsFileName && outputFileName->empty() ) "Unrecognized reporter, '" + reporterSpec.name() +
return ParserResult::runtimeError( "Reporter '" + reporterName + "' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." ); "'. Check available with --list-reporters" );
}
config.reporterSpecifications.push_back({ std::move(reporterName), std::move(outputFileName) });
// It would be enough to check this only once at the very end, but there is const bool hadOutputFile = reporterSpec.outputFile().some();
// not a place where we could call this check, so do it every time it could fail. config.reporterSpecifications.push_back( CATCH_MOVE( *parsed ) );
// For valid inputs, this is still called at most once. // It would be enough to check this only once at the very end, but
if (!containsFileName) { // there is not a place where we could call this check, so do it
// every time it could fail. For valid inputs, this is still called
// at most once.
if (!hadOutputFile) {
int n_reporters_without_file = 0; int n_reporters_without_file = 0;
for (auto const& spec : config.reporterSpecifications) { for (auto const& spec : config.reporterSpecifications) {
if (spec.outputFileName.none()) { if (spec.outputFile().none()) {
n_reporters_without_file++; n_reporters_without_file++;
} }
} }

4
src/catch2/reporters/catch_reporter_common_base.cpp
View File

@ -18,7 +18,9 @@ namespace Catch {
IEventListener( config.fullConfig() ), IEventListener( config.fullConfig() ),
m_wrapped_stream( config.stream() ), m_wrapped_stream( config.stream() ),
m_stream( m_wrapped_stream->stream() ), m_stream( m_wrapped_stream->stream() ),
m_colour( makeColourImpl( m_config->colourMode(), m_wrapped_stream ) ) {} m_colour( makeColourImpl( config.colourMode(), m_wrapped_stream ) ),
m_customOptions( config.customOptions() )
{}
ReporterBase::~ReporterBase() = default; ReporterBase::~ReporterBase() = default;

6
src/catch2/reporters/catch_reporter_common_base.hpp
View File

@ -10,6 +10,9 @@
#include <catch2/interfaces/catch_interfaces_reporter.hpp> #include <catch2/interfaces/catch_interfaces_reporter.hpp>
#include <map>
#include <string>
namespace Catch { namespace Catch {
class ColourImpl; class ColourImpl;
@ -30,7 +33,10 @@ namespace Catch {
//! Cached output stream from `m_wrapped_stream` to reduce //! Cached output stream from `m_wrapped_stream` to reduce
//! number of indirect calls needed to write output. //! number of indirect calls needed to write output.
std::ostream& m_stream; std::ostream& m_stream;
//! Colour implementation this reporter was configured for
Detail::unique_ptr<ColourImpl> m_colour; Detail::unique_ptr<ColourImpl> m_colour;
//! The custom reporter options user passed down to the reporter
std::map<std::string, std::string> m_customOptions;
public: public:
ReporterBase( ReporterConfig const& config ); ReporterBase( ReporterConfig const& config );

31
tests/CMakeLists.txt
View File

@ -435,8 +435,8 @@ if (NOT MEMORYCHECK_COMMAND)
add_test(NAME "MultiReporter::CapturingReportersDontPropagateStdOut" add_test(NAME "MultiReporter::CapturingReportersDontPropagateStdOut"
COMMAND COMMAND
$<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr" $<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr"
--reporter xml::${_NullFile} --reporter xml::out=${_NullFile}
--reporter junit::${_NullFile} --reporter junit::out=${_NullFile}
) )
set_tests_properties("MultiReporter::CapturingReportersDontPropagateStdOut" set_tests_properties("MultiReporter::CapturingReportersDontPropagateStdOut"
PROPERTIES PROPERTIES
@ -447,8 +447,8 @@ endif()
add_test(NAME "MultiReporter::NonCapturingReportersPropagateStdout" add_test(NAME "MultiReporter::NonCapturingReportersPropagateStdout"
COMMAND COMMAND
$<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr" $<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr"
--reporter xml::${_NullFile} --reporter xml::out=${_NullFile}
--reporter console::${_NullFile} --reporter console::out=${_NullFile}
) )
set_tests_properties("MultiReporter::NonCapturingReportersPropagateStdout" set_tests_properties("MultiReporter::NonCapturingReportersPropagateStdout"
PROPERTIES PROPERTIES
@ -468,13 +468,34 @@ set_tests_properties("Outputs::DashAsOutLocationSendsOutputToStdout"
add_test(NAME "Reporters::DashAsLocationInReporterSpecSendsOutputToStdout" add_test(NAME "Reporters::DashAsLocationInReporterSpecSendsOutputToStdout"
COMMAND COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed" $<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::- --reporter console::out=-
) )
set_tests_properties("Reporters::DashAsLocationInReporterSpecSendsOutputToStdout" set_tests_properties("Reporters::DashAsLocationInReporterSpecSendsOutputToStdout"
PROPERTIES PROPERTIES
PASS_REGULAR_EXPRESSION "All tests passed \\(5 assertions in 1 test case\\)" PASS_REGULAR_EXPRESSION "All tests passed \\(5 assertions in 1 test case\\)"
) )
add_test(NAME "Reporters::ReporterSpecificColourOverridesDefaultColour"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::colour-mode=ansi
--colour-mode none
)
set_tests_properties("Reporters::ReporterSpecificColourOverridesDefaultColour"
PROPERTIES
PASS_REGULAR_EXPRESSION "\\[1\;32mAll tests passed"
)
add_test(NAME "Reporters::UnrecognizedOptionInSpecCausesError"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::bad-option=ansi
)
set_tests_properties("Reporters::UnrecognizedOptionInSpecCausesError"
PROPERTIES
WILL_FAIL ON
)
add_test(NAME "Colours::ColourModeCanBeExplicitlySetToAnsi" add_test(NAME "Colours::ColourModeCanBeExplicitlySetToAnsi"
COMMAND COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed" $<TARGET_FILE:SelfTest> "Factorials are computed"

24
tests/ExtraTests/CMakeLists.txt
View File

@ -272,8 +272,8 @@ if (NOT MEMORYCHECK_COMMAND)
add_test( add_test(
NAME MultiReporter::NoncapturingListenerDoesntCauseStdoutPassThrough NAME MultiReporter::NoncapturingListenerDoesntCauseStdoutPassThrough
COMMAND ListenerStdoutCaptureInMultireporter COMMAND ListenerStdoutCaptureInMultireporter
--reporter xml::${_NullFile} --reporter xml::out=${_NullFile}
--reporter junit::${_NullFile} --reporter junit::out=${_NullFile}
) )
set_tests_properties( set_tests_properties(
@ -290,8 +290,8 @@ target_link_libraries(ListenerCanAskForCapturedStdout PRIVATE Catch2::Catch2With
add_test( add_test(
NAME MultiReporter::CapturingListenerCausesStdoutCapture NAME MultiReporter::CapturingListenerCausesStdoutCapture
COMMAND ListenerCanAskForCapturedStdout COMMAND ListenerCanAskForCapturedStdout
--reporter compact::${_NullFile} --reporter compact::out=${_NullFile}
--reporter console::${_NullFile} --reporter console::out=${_NullFile}
) )
set_tests_properties( set_tests_properties(
MultiReporter::CapturingListenerCausesStdoutCapture MultiReporter::CapturingListenerCausesStdoutCapture
@ -317,7 +317,7 @@ add_test(
NAME MultiReporter::PreferencesForPassingAssertionsIsRespected NAME MultiReporter::PreferencesForPassingAssertionsIsRespected
COMMAND ReporterPreferencesForPassingAssertionsIsRespected COMMAND ReporterPreferencesForPassingAssertionsIsRespected
--reporter test-reporter --reporter test-reporter
--reporter console::${_NullFile} --reporter console::out=${_NullFile}
) )
set_tests_properties( set_tests_properties(
MultiReporter::PreferencesForPassingAssertionsIsRespected MultiReporter::PreferencesForPassingAssertionsIsRespected
@ -339,6 +339,20 @@ set_tests_properties(
FAIL_REGULAR_EXPRESSION "X28 - ERROR" FAIL_REGULAR_EXPRESSION "X28 - ERROR"
) )
add_executable(CustomArgumentsForReporters ${TESTS_DIR}/X29-CustomArgumentsForReporters.cpp)
target_link_libraries(CustomArgumentsForReporters PRIVATE Catch2::Catch2WithMain)
add_test(
NAME CustomArgumentsForReporters
COMMAND CustomArgumentsForReporters
--reporter "test-reporter::Xa b=c 1::Xz:e = 1234"
)
set_tests_properties(
CustomArgumentsForReporters
PROPERTIES
PASS_REGULAR_EXPRESSION "Xa b=c 1::Xz:e = 1234"
)
add_executable(DuplicatedTestCases-SameNameAndTags ${TESTS_DIR}/X31-DuplicatedTestCases.cpp) add_executable(DuplicatedTestCases-SameNameAndTags ${TESTS_DIR}/X31-DuplicatedTestCases.cpp)
target_link_libraries(DuplicatedTestCases-SameNameAndTags PRIVATE Catch2::Catch2WithMain) target_link_libraries(DuplicatedTestCases-SameNameAndTags PRIVATE Catch2::Catch2WithMain)
add_test( add_test(

59
tests/ExtraTests/X29-CustomArgumentsForReporters.cpp Normal file
View File

@ -0,0 +1,59 @@
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE_1_0.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
/**\file
* Test that custom options are properly passed down to the reporter.
*
* We print out the arguments sorted by key, to have a stable expected
* output.
*/
#include <catch2/catch_test_macros.hpp>
#include <catch2/reporters/catch_reporter_streaming_base.hpp>
#include <catch2/reporters/catch_reporter_registrars.hpp>
#include <algorithm>
#include <iostream>
#include <string>
#include <utility>
#include <vector>
class TestReporter : public Catch::StreamingReporterBase {
public:
TestReporter( Catch::ReporterConfig const& _config ):
StreamingReporterBase( _config ) {
std::cout << "X29 - TestReporter constructed\n";
}
static std::string getDescription() {
return "X29 test reporter";
}
void testRunStarting( Catch::TestRunInfo const& ) override {
std::vector<std::pair<std::string, std::string>> options;
for ( auto const& kv : m_customOptions ) {
options.push_back( kv );
}
std::sort( options.begin(), options.end() );
bool first = true;
for ( auto const& kv : options ) {
if ( !first ) { std::cout << "::"; }
std::cout << kv.first << "=" << kv.second;
first = false;
}
std::cout << '\n';
}
~TestReporter() override;
};
TestReporter::~TestReporter() = default;
CATCH_REGISTER_REPORTER( "test-reporter", TestReporter )
TEST_CASE( "Just a test case to run things" ) {}

40
tests/SelfTest/Baselines/compact.sw.approved.txt
View File

@ -1316,10 +1316,8 @@ CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1 CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
==
{ console, <default-output> }
CmdLine.tests.cpp:<line number>: passed: result for: {?} CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false
@ -1333,37 +1331,21 @@ CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter" CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
== CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename" CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file." CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?} CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}

40
tests/SelfTest/Baselines/compact.sw.multi.approved.txt
View File

@ -1314,10 +1314,8 @@ CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1 CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
==
{ console, <default-output> }
CmdLine.tests.cpp:<line number>: passed: result for: {?} CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false
@ -1331,37 +1329,21 @@ CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter" CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
==
{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
== CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""' CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename" CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: !result for: true CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file." CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?} CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}

2
tests/SelfTest/Baselines/console.std.approved.txt
View File

@ -1396,5 +1396,5 @@ due to unexpected exception with message:
=============================================================================== ===============================================================================
test cases: 390 | 314 passed | 69 failed | 7 failed as expected test cases: 390 | 314 passed | 69 failed | 7 failed as expected
assertions: 2225 | 2070 passed | 128 failed | 27 failed as expected assertions: 2223 | 2068 passed | 128 failed | 27 failed as expected

76
tests/SelfTest/Baselines/console.sw.approved.txt
View File

@ -9433,16 +9433,14 @@ with expansion:
!false !false
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles().size() == 1 ) CHECK( cfg.getReporterSpecs().size() == 1 )
with expansion: with expansion:
1 == 1 1 == 1
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } ) CHECK( cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } )
with expansion: with expansion:
{ console, <default-output> } {?} == {?}
==
{ console, <default-output> }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -9544,11 +9542,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } )
with expansion: with expansion:
{ { console, <default-output> } } { {?} } == { {?} }
==
{ { console, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9568,11 +9564,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } )
with expansion: with expansion:
{ { xml, <default-output> } } { {?} } == { {?} }
==
{ { xml, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9592,11 +9586,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } )
with expansion: with expansion:
{ { junit, <default-output> } } { {?} } == { {?} }
==
{ { junit, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9635,11 +9627,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } )
with expansion: with expansion:
{ { console, out.txt } } { {?} } == { {?} }
==
{ { console, out.txt } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9659,34 +9649,12 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } )
with expansion: with expansion:
{ { console, C:\Temp\out.txt } } { {?} } == { {?} }
==
{ { console, C:\Temp\out.txt } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Output file cannot be empty
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("empty filename") )
with expansion:
"Reporter 'console' has empty filename specified as its output. Supply a
filename or remove the colons to use the default output." contains: "empty
filename"
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
reporter reporter
@ -9697,16 +9665,14 @@ CmdLine.tests.cpp:<line number>
............................................................................... ...............................................................................
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) ) CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) )
with expansion: with expansion:
{?} {?}
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } )
with expansion: with expansion:
{ { xml, output.xml }, { junit, output-junit.xml } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -9718,16 +9684,14 @@ CmdLine.tests.cpp:<line number>
............................................................................... ...............................................................................
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) ) CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) )
with expansion: with expansion:
{?} {?}
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } )
with expansion: with expansion:
{ { xml, output.xml }, { console, <default-output> } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { console, <default-output> } }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -17953,5 +17917,5 @@ Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 390 | 300 passed | 83 failed | 7 failed as expected test cases: 390 | 300 passed | 83 failed | 7 failed as expected
assertions: 2240 | 2070 passed | 143 failed | 27 failed as expected assertions: 2238 | 2068 passed | 143 failed | 27 failed as expected

76
tests/SelfTest/Baselines/console.sw.multi.approved.txt
View File

@ -9431,16 +9431,14 @@ with expansion:
!false !false
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles().size() == 1 ) CHECK( cfg.getReporterSpecs().size() == 1 )
with expansion: with expansion:
1 == 1 1 == 1
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } ) CHECK( cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } )
with expansion: with expansion:
{ console, <default-output> } {?} == {?}
==
{ console, <default-output> }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -9542,11 +9540,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } )
with expansion: with expansion:
{ { console, <default-output> } } { {?} } == { {?} }
==
{ { console, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9566,11 +9562,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } )
with expansion: with expansion:
{ { xml, <default-output> } } { {?} } == { {?} }
==
{ { xml, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9590,11 +9584,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } )
with expansion: with expansion:
{ { junit, <default-output> } } { {?} } == { {?} }
==
{ { junit, <default-output> } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9633,11 +9625,9 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } )
with expansion: with expansion:
{ { console, out.txt } } { {?} } == { {?} }
==
{ { console, out.txt } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
@ -9657,34 +9647,12 @@ with message:
result.errorMessage() := "" result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } )
with expansion: with expansion:
{ { console, C:\Temp\out.txt } } { {?} } == { {?} }
==
{ { console, C:\Temp\out.txt } }
with message: with message:
result.errorMessage() := "" result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Output file cannot be empty
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("empty filename") )
with expansion:
"Reporter 'console' has empty filename specified as its output. Supply a
filename or remove the colons to use the default output." contains: "empty
filename"
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
reporter reporter
@ -9695,16 +9663,14 @@ CmdLine.tests.cpp:<line number>
............................................................................... ...............................................................................
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) ) CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) )
with expansion: with expansion:
{?} {?}
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } )
with expansion: with expansion:
{ { xml, output.xml }, { junit, output-junit.xml } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -9716,16 +9682,14 @@ CmdLine.tests.cpp:<line number>
............................................................................... ...............................................................................
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) ) CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) )
with expansion: with expansion:
{?} {?}
CmdLine.tests.cpp:<line number>: PASSED: CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } ) REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } )
with expansion: with expansion:
{ { xml, output.xml }, { console, <default-output> } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { console, <default-output> } }
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Process can be configured on command line Process can be configured on command line
@ -17945,5 +17909,5 @@ Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 390 | 300 passed | 83 failed | 7 failed as expected test cases: 390 | 300 passed | 83 failed | 7 failed as expected
assertions: 2240 | 2070 passed | 143 failed | 27 failed as expected assertions: 2238 | 2068 passed | 143 failed | 27 failed as expected

3
tests/SelfTest/Baselines/junit.sw.approved.txt
View File

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact <testsuitesloose text artifact
> >
<testsuite name="<exe-name>" errors="17" failures="126" tests="2240" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="126" tests="2238" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties> <properties>
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/> <property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@ -1109,7 +1109,6 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Output file cannot be empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>

3
tests/SelfTest/Baselines/junit.sw.multi.approved.txt
View File

@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<testsuites> <testsuites>
<testsuite name="<exe-name>" errors="17" failures="126" tests="2240" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="126" tests="2238" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties> <properties>
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/> <property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@ -1108,7 +1108,6 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Output file cannot be empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>

1
tests/SelfTest/Baselines/sonarqube.sw.approved.txt
View File

@ -66,7 +66,6 @@
<testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Output file cannot be empty" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/>

1
tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt
View File

@ -65,7 +65,6 @@
<testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Output file cannot be empty" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/> <testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/>

28
tests/SelfTest/Baselines/tap.sw.approved.txt
View File

@ -2479,9 +2479,9 @@ ok {test-number} - config.reporterSpecifications.empty() for: true
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !(cfg.hasTestFilters()) for: !false ok {test-number} - !(cfg.hasTestFilters()) for: !false
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1 ok {test-number} - cfg.getReporterSpecs().size() == 1 for: 1 == 1
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } == { console, <default-output> } ok {test-number} - cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} ok {test-number} - result for: {?}
# Process can be configured on command line # Process can be configured on command line
@ -2509,15 +2509,15 @@ ok {test-number} - cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for:
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } == { { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } == { { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - !result for: true
# Process can be configured on command line # Process can be configured on command line
@ -2525,23 +2525,19 @@ ok {test-number} - result.errorMessage(), ContainsSubstring("Unrecognized report
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } == { { console, out.txt } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } == { { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename" ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?} ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } } == { { xml, output.xml }, { junit, output-junit.xml } } ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } } == { { xml, output.xml }, { console, <default-output> } }
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - !result for: true
# Process can be configured on command line # Process can be configured on command line
@ -4482,5 +4478,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} - ok {test-number} -
# xmlentitycheck # xmlentitycheck
ok {test-number} - ok {test-number} -
1..2240 1..2238

28
tests/SelfTest/Baselines/tap.sw.multi.approved.txt
View File

@ -2477,9 +2477,9 @@ ok {test-number} - config.reporterSpecifications.empty() for: true
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !(cfg.hasTestFilters()) for: !false ok {test-number} - !(cfg.hasTestFilters()) for: !false
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1 ok {test-number} - cfg.getReporterSpecs().size() == 1 for: 1 == 1
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } == { console, <default-output> } ok {test-number} - cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} ok {test-number} - result for: {?}
# Process can be configured on command line # Process can be configured on command line
@ -2507,15 +2507,15 @@ ok {test-number} - cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for:
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } == { { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } == { { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - !result for: true
# Process can be configured on command line # Process can be configured on command line
@ -2523,23 +2523,19 @@ ok {test-number} - result.errorMessage(), ContainsSubstring("Unrecognized report
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } == { { console, out.txt } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""' ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } == { { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""' ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename" ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?} ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } } == { { xml, output.xml }, { junit, output-junit.xml } } ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } } == { { xml, output.xml }, { console, <default-output> } }
# Process can be configured on command line # Process can be configured on command line
ok {test-number} - !result for: true ok {test-number} - !result for: true
# Process can be configured on command line # Process can be configured on command line
@ -4474,5 +4470,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} - ok {test-number} -
# xmlentitycheck # xmlentitycheck
ok {test-number} - ok {test-number} -
1..2240 1..2238

78
tests/SelfTest/Baselines/xml.sw.approved.txt
View File

@ -11468,7 +11468,7 @@ C
</Expression> </Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cfg.getReportersAndOutputFiles().size() == 1 cfg.getReporterSpecs().size() == 1
</Original> </Original>
<Expanded> <Expanded>
1 == 1 1 == 1
@ -11476,12 +11476,10 @@ C
</Expression> </Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} }
</Original> </Original>
<Expanded> <Expanded>
{ console, &lt;default-output> } {?} == {?}
==
{ console, &lt;default-output> }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="9" failures="0" expectedFailures="0"/> <OverallResults successes="9" failures="0" expectedFailures="0"/>
@ -11618,12 +11616,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, &lt;default-output> } } { {?} } == { {?} }
==
{ { console, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11648,12 +11644,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, &lt;default-output> } } { {?} } == { {?} }
==
{ { xml, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11678,12 +11672,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { junit, &lt;default-output> } } { {?} } == { {?} }
==
{ { junit, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11730,12 +11722,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, out.txt } } { {?} } == { {?} }
==
{ { console, out.txt } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11760,34 +11750,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, C:\Temp\out.txt } } { {?} } == { {?} }
==
{ { console, C:\Temp\out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Output file cannot be empty" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("empty filename")
</Original>
<Expanded>
"Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11799,7 +11765,7 @@ C
<Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" })
</Original> </Original>
<Expanded> <Expanded>
{?} {?}
@ -11807,12 +11773,10 @@ C
</Expression> </Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, output.xml }, { junit, output-junit.xml } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11826,7 +11790,7 @@ C
<Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" })
</Original> </Original>
<Expanded> <Expanded>
{?} {?}
@ -11834,12 +11798,10 @@ C
</Expression> </Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, output.xml }, { console, &lt;default-output> } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { console, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -21072,6 +21034,6 @@ loose text artifact
</Section> </Section>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<OverallResults successes="2070" failures="143" expectedFailures="27"/> <OverallResults successes="2068" failures="143" expectedFailures="27"/>
<OverallResultsCases successes="300" failures="83" expectedFailures="7"/> <OverallResultsCases successes="300" failures="83" expectedFailures="7"/>
</Catch2TestRun> </Catch2TestRun>

78
tests/SelfTest/Baselines/xml.sw.multi.approved.txt
View File

@ -11468,7 +11468,7 @@ C
</Expression> </Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cfg.getReportersAndOutputFiles().size() == 1 cfg.getReporterSpecs().size() == 1
</Original> </Original>
<Expanded> <Expanded>
1 == 1 1 == 1
@ -11476,12 +11476,10 @@ C
</Expression> </Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} }
</Original> </Original>
<Expanded> <Expanded>
{ console, &lt;default-output> } {?} == {?}
==
{ console, &lt;default-output> }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="9" failures="0" expectedFailures="0"/> <OverallResults successes="9" failures="0" expectedFailures="0"/>
@ -11618,12 +11616,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, &lt;default-output> } } { {?} } == { {?} }
==
{ { console, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11648,12 +11644,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, &lt;default-output> } } { {?} } == { {?} }
==
{ { xml, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11678,12 +11672,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { junit, &lt;default-output> } } { {?} } == { {?} }
==
{ { junit, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11730,12 +11722,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, out.txt } } { {?} } == { {?} }
==
{ { console, out.txt } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11760,34 +11750,10 @@ C
</Info> </Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { console, C:\Temp\out.txt } } { {?} } == { {?} }
==
{ { console, C:\Temp\out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Output file cannot be empty" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("empty filename")
</Original>
<Expanded>
"Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11799,7 +11765,7 @@ C
<Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" })
</Original> </Original>
<Expanded> <Expanded>
{?} {?}
@ -11807,12 +11773,10 @@ C
</Expression> </Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, output.xml }, { junit, output-junit.xml } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11826,7 +11790,7 @@ C
<Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" })
</Original> </Original>
<Expanded> <Expanded>
{?} {?}
@ -11834,12 +11798,10 @@ C
</Expression> </Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" > <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original> <Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } }
</Original> </Original>
<Expanded> <Expanded>
{ { xml, output.xml }, { console, &lt;default-output> } } { {?}, {?} } == { {?}, {?} }
==
{ { xml, output.xml }, { console, &lt;default-output> } }
</Expanded> </Expanded>
</Expression> </Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/> <OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -21071,6 +21033,6 @@ There is no extra whitespace here
</Section> </Section>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<OverallResults successes="2070" failures="143" expectedFailures="27"/> <OverallResults successes="2068" failures="143" expectedFailures="27"/>
<OverallResultsCases successes="300" failures="83" expectedFailures="7"/> <OverallResultsCases successes="300" failures="83" expectedFailures="7"/>
</Catch2TestRun> </Catch2TestRun>

47
tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp
View File

@ -361,9 +361,9 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
#endif #endif
; ;
CHECK( cfg.getReportersAndOutputFiles().size() == 1 ); CHECK( cfg.getReporterSpecs().size() == 1 );
CHECK( cfg.getReportersAndOutputFiles()[0] == CHECK( cfg.getReporterSpecs()[0] ==
Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } ); Catch::ReporterSpec{ expectedReporter, {}, {}, {} } );
} }
SECTION("test lists") { SECTION("test lists") {
@ -399,28 +399,31 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
} }
SECTION("reporter") { SECTION("reporter") {
using vec_ReporterAndFile = std::vector<Catch::ConfigData::ReporterAndFile>; using vec_Specs = std::vector<Catch::ReporterSpec>;
using namespace std::string_literals; using namespace std::string_literals;
SECTION("-r/console") { SECTION("-r/console") {
auto result = cli.parse({"test", "-r", "console"}); auto result = cli.parse({"test", "-r", "console"});
CAPTURE(result.errorMessage()); CAPTURE(result.errorMessage());
CHECK(result); CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "console", {}, {}, {} } } );
} }
SECTION("-r/xml") { SECTION("-r/xml") {
auto result = cli.parse({"test", "-r", "xml"}); auto result = cli.parse({"test", "-r", "xml"});
CAPTURE(result.errorMessage()); CAPTURE(result.errorMessage());
CHECK(result); CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", {}, {}, {} } } );
} }
SECTION("--reporter/junit") { SECTION("--reporter/junit") {
auto result = cli.parse({"test", "--reporter", "junit"}); auto result = cli.parse({"test", "--reporter", "junit"});
CAPTURE(result.errorMessage()); CAPTURE(result.errorMessage());
CHECK(result); CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "junit", {}, {}, {} } } );
} }
SECTION("must match one of the available ones") { SECTION("must match one of the available ones") {
auto result = cli.parse({"test", "--reporter", "unsupported"}); auto result = cli.parse({"test", "--reporter", "unsupported"});
@ -429,34 +432,34 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Unrecognized reporter")); REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Unrecognized reporter"));
} }
SECTION("With output file") { SECTION("With output file") {
auto result = cli.parse({ "test", "-r", "console::out.txt" }); auto result = cli.parse({ "test", "-r", "console::out=out.txt" });
CAPTURE(result.errorMessage()); CAPTURE(result.errorMessage());
CHECK(result); CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "console", "out.txt"s, {}, {} } } );
} }
SECTION("With Windows-like absolute path as output file") { SECTION("With Windows-like absolute path as output file") {
auto result = cli.parse({ "test", "-r", "console::C:\\Temp\\out.txt" }); auto result = cli.parse({ "test", "-r", "console::out=C:\\Temp\\out.txt" });
CAPTURE(result.errorMessage()); CAPTURE(result.errorMessage());
CHECK(result); CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} }); REQUIRE( config.reporterSpecifications ==
} vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } );
SECTION("Output file cannot be empty") {
auto result = cli.parse({"test", "--reporter", "console::"});
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("empty filename"));
} }
SECTION("Multiple reporters") { SECTION("Multiple reporters") {
SECTION("All with output files") { SECTION("All with output files") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" })); CHECK(cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", "output.xml"s, {}, {} },
{ "junit", "output-junit.xml"s, {}, {} } } );
} }
SECTION("Mixed output files and default output") { SECTION("Mixed output files and default output") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" })); CHECK(cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} }); REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", "output.xml"s, {}, {} },
{ "console", {}, {}, {} } } );
} }
SECTION("cannot have multiple reporters with default output") { SECTION("cannot have multiple reporters with default output") {
auto result = cli.parse({ "test", "-r", "console", "-r", "xml::output.xml", "-r", "junit" }); auto result = cli.parse({ "test", "-r", "console", "-r", "xml::out=output.xml", "-r", "junit" });
CHECK(!result); CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.")); REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file."));
} }

9
tests/SelfTest/IntrospectiveTests/Reporters.tests.cpp
View File

@ -89,7 +89,8 @@ TEST_CASE( "Reporter's write listings to provided stream", "[reporters]" ) {
Catch::ConfigData config_data; Catch::ConfigData config_data;
Catch::Config config( config_data ); Catch::Config config( config_data );
Catch::ReporterConfig rep_config( &config, &sstream ); Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
auto reporter = factory.second->create( rep_config ); auto reporter = factory.second->create( rep_config );
DYNAMIC_SECTION( factory.first << " reporter lists tags" ) { DYNAMIC_SECTION( factory.first << " reporter lists tags" ) {
@ -176,7 +177,8 @@ TEST_CASE("Multireporter calls reporters and listeners in correct order",
Catch::ConfigData config_data; Catch::ConfigData config_data;
Catch::Config config( config_data ); Catch::Config config( config_data );
StringIStream sstream; StringIStream sstream;
Catch::ReporterConfig rep_config( &config, &sstream ); Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
// We add reporters before listeners, to check that internally they // We add reporters before listeners, to check that internally they
// get sorted properly, and listeners are called first anyway. // get sorted properly, and listeners are called first anyway.
@ -229,7 +231,8 @@ TEST_CASE("Multireporter updates ReporterPreferences properly",
Catch::ConfigData config_data; Catch::ConfigData config_data;
Catch::Config config( config_data ); Catch::Config config( config_data );
StringIStream sstream; StringIStream sstream;
Catch::ReporterConfig rep_config( &config, &sstream ); Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
Catch::MultiReporter multiReporter( &config ); Catch::MultiReporter multiReporter( &config );
// Post init defaults // Post init defaults

2
tools/scripts/approvalTests.py
View File

@ -220,7 +220,7 @@ common_args = ["~[!nonportable]~[!benchmark]~[approvals] *", "-s", "-w", "NoAsse
filenames = ['{}.sw.multi'.format(reporter) for reporter in reporters] filenames = ['{}.sw.multi'.format(reporter) for reporter in reporters]
reporter_args = [] reporter_args = []
for reporter, filename in zip(reporters, filenames): for reporter, filename in zip(reporters, filenames):
reporter_args += ['-r', '{}::{}'.format(reporter, get_rawResultsPath(filename))] reporter_args += ['-r', '{}::out={}'.format(reporter, get_rawResultsPath(filename))]
run_test("default.sw.multi", common_args + reporter_args) run_test("default.sw.multi", common_args + reporter_args)
check_outputs("default.sw.multi") check_outputs("default.sw.multi")