3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-24 22:36:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

Integrate the new reporter specs into Catch2

Browse Source 
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.

Closes #339 as it allows per-reporter forcing of ansi colour codes.
This commit is contained in:
Martin Hořeňovský 2022-04-04 21:07:04 +02:00
parent 3c06bcb374
commit 423e1d2ebb
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
29 changed files with 395 additions and 448 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
docs/command-line.md
View File

@ -123,22 +123,33 @@ Test names containing special characters, such as `,` or `[` can specify them on
<a id="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use
<pre>-r, --reporter &lt;reporter[::output-file]&gt;</pre>
> Support for providing output-file through the `-r`, `--reporter` flag was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
<pre>-r, --reporter &lt;reporter[::key=value]*&gt;</pre>
Reporters are how the output from Catch2 (results of assertions, tests,
benchmarks and so on) is formatted and written out. The default reporter
is called the "Console" reporter and is intended to provide relatively
verbose and human-friendly output.
Reporters are also individually configurable. To pass configuration options
to the reporter, you append `::key=value` to the reporter specification
as many times as you want, e.g. `--reporter xml::out=someFile.xml`.
The keys must either be prefixed by "X", in which case they are not parsed
by Catch2 and are only passed down to the reporter, or one of options
hardcoded into Catch2. Currently there are only 2,
["out"](#sending-output-to-a-file), and ["colour-mode"](#colour-mode).
_Note that the reporter might still check the X-prefixed options for
validity, and throw an error if they are wrong._
> Support for passing arguments to reporters through the `-r`, `--reporter` flag was introduced in Catch2 X.Y.Z
There are multiple built-in reporters, you can see what they do by using the
[`--list-reporter`](command-line.md#listing-available-tests-tags-or-reporters)
flag. If you need a reporter providing custom format outside of the already
provided ones, look at the ["write your own reporter" part of the reporter
documentation](reporters.md#writing-your-own-reporter).
This option may be passed multiple times to use multiple (different)
reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters)
for details on what the resulting behaviour is. Also note that at most one
@ -148,13 +159,12 @@ the [`-o`, `--out`](#sending-output-to-a-file) option.
> Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z
As with the `--out` option, using `-` for the output file name sends the
output to stdout.
_Note: There is currently no way to escape `::` in the reporter spec,
and thus reporter/file names with `::` in them will not work properly.
As `::` in paths is relatively obscure (unlike `:`), we do not consider
this an issue._
and thus the reporter names, or configuration keys and values, cannot
contain `::`. As `::` in paths is relatively obscure (unlike ':'), we do
not consider this an issue._
<a id="breaking-into-the-debugger"></a>
## Breaking into the debugger

27
docs/reporters.md
View File

@ -43,9 +43,14 @@ them write into different destinations. The two main uses of this are
Specifying multiple reporter looks like this:
```
--reporter console::- --reporter JUnit::result-junit.xml
--reporter JUnit::out=result-junit.xml --reporter console::out=-::colour-mode=ansi
```
This tells Catch2 to use two reporters, `JUnit` reporter that writes
its machine-readable XML output to file `result-junit.xml`, and the
`console` reporter that writes its user-friendly output to stdout and
uses ANSI colour codes for colouring the output.
Using multiple reporters (or one reporter and one-or-more [event
listeners](event-listener.md#top)) can have surprisingly complex semantics
when using customization points provided to reporters by Catch2, namely
@ -162,6 +167,26 @@ Currently there are two customization options:
format includes passing assertions even without the `-s` flag.
### Per-reporter configuration
> Per-reporter configuration was introduced in Catch2 X.Y.Z
Catch2 supports some configuration to happen per reporter. The configuration
options fall into one of two categories:
* Catch2-recognized options
* Reporter-specific options
The former is a small set of universal options that Catch2 handles for
the reporters, e.g. output file or console colour mode. The latter are
options that the reporters have to handle themselves, but the keys and
values can be arbitrary strings, as long as they don't contain `::`. This
allows writing reporters that can be significantly customized at runtime.
Reporter-specific options always have to be prefixed with "X" (large
letter X).
### Other expected functionality of a reporter
When writing a custom reporter, there are few more things that you should

21
src/catch2/catch_config.cpp
View File

@ -33,17 +33,6 @@ namespace Catch {
} // unnamed namespace
} // namespace Detail
std::ostream& operator<<( std::ostream& os,
ConfigData::ReporterAndFile const& reporter ) {
os << "{ " << reporter.reporterName << ", ";
if ( reporter.outputFileName ) {
os << *reporter.outputFileName;
} else {
os << "<default-output>";
}
return os << " }";
}
Config::Config( ConfigData const& data ):
m_data( data ) {
// We need to trim filter specs to avoid trouble with superfluous
@ -76,14 +65,14 @@ namespace Catch {
#else
"console",
#endif
{}
{}, {}, {}
} );
}
bool defaultOutputUsed = false;
m_reporterStreams.reserve( m_data.reporterSpecifications.size() );
for ( auto const& reporterAndFile : m_data.reporterSpecifications ) {
if ( reporterAndFile.outputFileName.none() ) {
for ( auto const& reporterSpec : m_data.reporterSpecifications ) {
if ( reporterSpec.outputFile().none() ) {
CATCH_ENFORCE( !defaultOutputUsed,
"Internal error: cannot use default output for "
"multiple reporters" );
@ -93,7 +82,7 @@ namespace Catch {
openStream( data.defaultOutputFilename ) );
} else {
m_reporterStreams.push_back(
openStream( *reporterAndFile.outputFileName ) );
openStream( *reporterSpec.outputFile() ) );
}
}
}
@ -108,7 +97,7 @@ namespace Catch {
std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; }
std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; }
std::vector<ConfigData::ReporterAndFile> const& Config::getReportersAndOutputFiles() const {
std::vector<ReporterSpec> const& Config::getReporterSpecs() const {
return m_data.reporterSpecifications;
}

16
src/catch2/catch_config.hpp
View File

@ -14,6 +14,7 @@
#include <catch2/internal/catch_unique_ptr.hpp>
#include <catch2/internal/catch_optional.hpp>
#include <catch2/internal/catch_random_seed_generation.hpp>
#include <catch2/internal/catch_reporter_spec_parser.hpp>
#include <iosfwd>
#include <vector>
@ -24,17 +25,6 @@ namespace Catch {
class IStream;
struct ConfigData {
struct ReporterAndFile {
std::string reporterName;
// If none, the output goes to the default output.
Optional<std::string> outputFileName;
friend bool operator==(ReporterAndFile const& lhs, ReporterAndFile const& rhs) {
return lhs.reporterName == rhs.reporterName && lhs.outputFileName == rhs.outputFileName;
}
friend std::ostream& operator<<(std::ostream &os, ReporterAndFile const& reporter);
};
bool listTests = false;
bool listTags = false;
@ -72,7 +62,7 @@ namespace Catch {
std::string defaultOutputFilename;
std::string name;
std::string processName;
std::vector<ReporterAndFile> reporterSpecifications;
std::vector<ReporterSpec> reporterSpecifications;
std::vector<std::string> testsOrTags;
std::vector<std::string> sectionsToRun;
@ -90,7 +80,7 @@ namespace Catch {
bool listTags() const;
bool listReporters() const;
std::vector<ConfigData::ReporterAndFile> const& getReportersAndOutputFiles() const;
std::vector<ReporterSpec> const& getReporterSpecs() const;
IStream const* getReporterOutputStream(std::size_t reporterIdx) const;
std::vector<std::string> const& getTestsOrTags() const override;

25
src/catch2/catch_session.cpp
View File

@ -41,11 +41,18 @@ namespace Catch {
return reporter;
}
IStreamingReporterPtr makeReporter(Config const* config) {
IStreamingReporterPtr prepareReporters(Config const* config) {
if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty()
&& config->getReportersAndOutputFiles().size() == 1) {
&& config->getReporterSpecs().size() == 1) {
auto const& spec = config->getReporterSpecs()[0];
auto stream = config->getReporterOutputStream(0);
return createReporter(config->getReportersAndOutputFiles()[0].reporterName, ReporterConfig(config, stream));
return createReporter(
config->getReporterSpecs()[0].name(),
ReporterConfig(
config,
stream,
spec.colourMode().valueOr( config->colourMode() ),
spec.customOptions() ) );
}
auto multi = Detail::make_unique<MultiReporter>(config);
@ -56,9 +63,15 @@ namespace Catch {
}
std::size_t reporterIdx = 0;
for (auto const& reporterAndFile : config->getReportersAndOutputFiles()) {
for (auto const& reporterSpec : config->getReporterSpecs()) {
auto stream = config->getReporterOutputStream(reporterIdx);
multi->addReporter(createReporter(reporterAndFile.reporterName, ReporterConfig(config, stream)));
multi->addReporter( createReporter(
reporterSpec.name(),
ReporterConfig( config,
stream,
reporterSpec.colourMode().valueOr(
config->colourMode() ),
reporterSpec.customOptions() ) ) );
reporterIdx++;
}
@ -304,7 +317,7 @@ namespace Catch {
getCurrentMutableContext().setConfig(m_config.get());
// Create reporter(s) so we can route listings through them
auto reporter = makeReporter(m_config.get());
auto reporter = prepareReporters(m_config.get());
auto const& invalidSpecs = m_config->testSpec().getInvalidSpecs();
if ( !invalidSpecs.empty() ) {

18
src/catch2/interfaces/catch_interfaces_reporter.cpp
View File

@ -14,17 +14,31 @@
#include <catch2/internal/catch_string_manip.hpp>
#include <catch2/catch_test_case_info.hpp>
#include <catch2/reporters/catch_reporter_helpers.hpp>
#include <catch2/internal/catch_move_and_forward.hpp>
#include <algorithm>
#include <iomanip>
namespace Catch {
ReporterConfig::ReporterConfig( IConfig const* _fullConfig, IStream const* _stream )
: m_stream( _stream ), m_fullConfig( _fullConfig ) {}
ReporterConfig::ReporterConfig(
IConfig const* _fullConfig,
IStream const* _stream,
ColourMode colourMode,
std::map<std::string, std::string> customOptions ):
m_stream( _stream ),
m_fullConfig( _fullConfig ),
m_colourMode( colourMode ),
m_customOptions( CATCH_MOVE( customOptions ) ) {}
IStream const* ReporterConfig::stream() const { return m_stream; }
IConfig const * ReporterConfig::fullConfig() const { return m_fullConfig; }
ColourMode ReporterConfig::colourMode() const { return m_colourMode; }
std::map<std::string, std::string> const&
ReporterConfig::customOptions() const {
return m_customOptions;
}
AssertionStats::AssertionStats( AssertionResult const& _assertionResult,
std::vector<MessageInfo> const& _infoMessages,

11
src/catch2/interfaces/catch_interfaces_reporter.hpp
View File

@ -19,6 +19,7 @@
#include <catch2/benchmark/catch_outlier_classification.hpp>
#include <map>
#include <string>
#include <vector>
#include <iosfwd>
@ -31,16 +32,24 @@ namespace Catch {
class TestCaseHandle;
struct IConfig;
class IStream;
enum class ColourMode : std::uint8_t;
struct ReporterConfig {
ReporterConfig( IConfig const* _fullConfig, IStream const* _stream );
ReporterConfig( IConfig const* _fullConfig,
IStream const* _stream,
ColourMode colourMode,
std::map<std::string, std::string> customOptions );
IStream const* stream() const;
IConfig const* fullConfig() const;
ColourMode colourMode() const;
std::map<std::string, std::string> const& customOptions() const;
private:
IStream const* m_stream;
IConfig const* m_fullConfig;
ColourMode m_colourMode;
std::map<std::string, std::string> m_customOptions;
};
struct TestRunInfo {

61
src/catch2/internal/catch_commandline.cpp
View File

@ -141,57 +141,42 @@ namespace Catch {
return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' );
return ParserResult::ok( ParseResultType::Matched );
};
auto const setReporter = [&]( std::string const& reporterSpec ) {
if ( reporterSpec.empty() ) {
auto const setReporter = [&]( std::string const& userReporterSpec ) {
if ( userReporterSpec.empty() ) {
return ParserResult::runtimeError( "Received empty reporter spec." );
}
// Exactly one of the reporters may be specified without an output
// file, in which case it defaults to the output specified by "-o"
// (or standard output).
static constexpr auto separator = "::";
static constexpr size_t separatorSize = 2;
auto fileNameSeparatorPos = reporterSpec.find( separator );
const bool containsFileName = fileNameSeparatorPos != reporterSpec.npos;
if ( containsFileName ) {
auto nextSeparatorPos = reporterSpec.find(
separator, fileNameSeparatorPos + separatorSize );
if ( nextSeparatorPos != reporterSpec.npos ) {
return ParserResult::runtimeError(
"Too many separators in reporter spec '" + reporterSpec + '\'' );
}
Optional<ReporterSpec> parsed =
parseReporterSpec( userReporterSpec );
if ( !parsed ) {
return ParserResult::runtimeError(
"Could not parse reporter spec '" + userReporterSpec +
"'" );
}
std::string reporterName;
Optional<std::string> outputFileName;
reporterName = reporterSpec.substr( 0, fileNameSeparatorPos );
if ( reporterName.empty() ) {
return ParserResult::runtimeError( "Reporter name cannot be empty." );
}
if ( containsFileName ) {
outputFileName = reporterSpec.substr(
fileNameSeparatorPos + separatorSize, reporterSpec.size() );
}
auto const& reporterSpec = *parsed;
IReporterRegistry::FactoryMap const& factories =
getRegistryHub().getReporterRegistry().getFactories();
auto result = factories.find( reporterName );
auto result = factories.find( reporterSpec.name() );
if( result == factories.end() )
return ParserResult::runtimeError( "Unrecognized reporter, '" + reporterName + "'. Check available with --list-reporters" );
if( containsFileName && outputFileName->empty() )
return ParserResult::runtimeError( "Reporter '" + reporterName + "' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." );
if ( result == factories.end() ) {
return ParserResult::runtimeError(
"Unrecognized reporter, '" + reporterSpec.name() +
"'. Check available with --list-reporters" );
}
config.reporterSpecifications.push_back({ std::move(reporterName), std::move(outputFileName) });
// It would be enough to check this only once at the very end, but there is
// not a place where we could call this check, so do it every time it could fail.
// For valid inputs, this is still called at most once.
if (!containsFileName) {
const bool hadOutputFile = reporterSpec.outputFile().some();
config.reporterSpecifications.push_back( CATCH_MOVE( *parsed ) );
// It would be enough to check this only once at the very end, but
// there is not a place where we could call this check, so do it
// every time it could fail. For valid inputs, this is still called
// at most once.
if (!hadOutputFile) {
int n_reporters_without_file = 0;
for (auto const& spec : config.reporterSpecifications) {
if (spec.outputFileName.none()) {
if (spec.outputFile().none()) {
n_reporters_without_file++;
}
}

4
src/catch2/reporters/catch_reporter_common_base.cpp
View File

@ -18,7 +18,9 @@ namespace Catch {
IEventListener( config.fullConfig() ),
m_wrapped_stream( config.stream() ),
m_stream( m_wrapped_stream->stream() ),
m_colour( makeColourImpl( m_config->colourMode(), m_wrapped_stream ) ) {}
m_colour( makeColourImpl( config.colourMode(), m_wrapped_stream ) ),
m_customOptions( config.customOptions() )
{}
ReporterBase::~ReporterBase() = default;

6
src/catch2/reporters/catch_reporter_common_base.hpp
View File

@ -10,6 +10,9 @@
#include <catch2/interfaces/catch_interfaces_reporter.hpp>
#include <map>
#include <string>
namespace Catch {
class ColourImpl;
@ -30,7 +33,10 @@ namespace Catch {
//! Cached output stream from `m_wrapped_stream` to reduce
//! number of indirect calls needed to write output.
std::ostream& m_stream;
//! Colour implementation this reporter was configured for
Detail::unique_ptr<ColourImpl> m_colour;
//! The custom reporter options user passed down to the reporter
std::map<std::string, std::string> m_customOptions;
public:
ReporterBase( ReporterConfig const& config );

31
tests/CMakeLists.txt
View File

@ -435,8 +435,8 @@ if (NOT MEMORYCHECK_COMMAND)
add_test(NAME "MultiReporter::CapturingReportersDontPropagateStdOut"
COMMAND
$<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr"
--reporter xml::${_NullFile}
--reporter junit::${_NullFile}
--reporter xml::out=${_NullFile}
--reporter junit::out=${_NullFile}
)
set_tests_properties("MultiReporter::CapturingReportersDontPropagateStdOut"
PROPERTIES
@ -447,8 +447,8 @@ endif()
add_test(NAME "MultiReporter::NonCapturingReportersPropagateStdout"
COMMAND
$<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr"
--reporter xml::${_NullFile}
--reporter console::${_NullFile}
--reporter xml::out=${_NullFile}
--reporter console::out=${_NullFile}
)
set_tests_properties("MultiReporter::NonCapturingReportersPropagateStdout"
PROPERTIES
@ -468,13 +468,34 @@ set_tests_properties("Outputs::DashAsOutLocationSendsOutputToStdout"
add_test(NAME "Reporters::DashAsLocationInReporterSpecSendsOutputToStdout"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::-
--reporter console::out=-
)
set_tests_properties("Reporters::DashAsLocationInReporterSpecSendsOutputToStdout"
PROPERTIES
PASS_REGULAR_EXPRESSION "All tests passed \\(5 assertions in 1 test case\\)"
)
add_test(NAME "Reporters::ReporterSpecificColourOverridesDefaultColour"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::colour-mode=ansi
--colour-mode none
)
set_tests_properties("Reporters::ReporterSpecificColourOverridesDefaultColour"
PROPERTIES
PASS_REGULAR_EXPRESSION "\\[1\;32mAll tests passed"
)
add_test(NAME "Reporters::UnrecognizedOptionInSpecCausesError"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
--reporter console::bad-option=ansi
)
set_tests_properties("Reporters::UnrecognizedOptionInSpecCausesError"
PROPERTIES
WILL_FAIL ON
)
add_test(NAME "Colours::ColourModeCanBeExplicitlySetToAnsi"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"

24
tests/ExtraTests/CMakeLists.txt
View File

@ -272,8 +272,8 @@ if (NOT MEMORYCHECK_COMMAND)
add_test(
NAME MultiReporter::NoncapturingListenerDoesntCauseStdoutPassThrough
COMMAND ListenerStdoutCaptureInMultireporter
--reporter xml::${_NullFile}
--reporter junit::${_NullFile}
--reporter xml::out=${_NullFile}
--reporter junit::out=${_NullFile}
)
set_tests_properties(
@ -290,8 +290,8 @@ target_link_libraries(ListenerCanAskForCapturedStdout PRIVATE Catch2::Catch2With
add_test(
NAME MultiReporter::CapturingListenerCausesStdoutCapture
COMMAND ListenerCanAskForCapturedStdout
--reporter compact::${_NullFile}
--reporter console::${_NullFile}
--reporter compact::out=${_NullFile}
--reporter console::out=${_NullFile}
)
set_tests_properties(
MultiReporter::CapturingListenerCausesStdoutCapture
@ -317,7 +317,7 @@ add_test(
NAME MultiReporter::PreferencesForPassingAssertionsIsRespected
COMMAND ReporterPreferencesForPassingAssertionsIsRespected
--reporter test-reporter
--reporter console::${_NullFile}
--reporter console::out=${_NullFile}
)
set_tests_properties(
MultiReporter::PreferencesForPassingAssertionsIsRespected
@ -339,6 +339,20 @@ set_tests_properties(
FAIL_REGULAR_EXPRESSION "X28 - ERROR"
)
add_executable(CustomArgumentsForReporters ${TESTS_DIR}/X29-CustomArgumentsForReporters.cpp)
target_link_libraries(CustomArgumentsForReporters PRIVATE Catch2::Catch2WithMain)
add_test(
NAME CustomArgumentsForReporters
COMMAND CustomArgumentsForReporters
--reporter "test-reporter::Xa b=c 1::Xz:e = 1234"
)
set_tests_properties(
CustomArgumentsForReporters
PROPERTIES
PASS_REGULAR_EXPRESSION "Xa b=c 1::Xz:e = 1234"
)
add_executable(DuplicatedTestCases-SameNameAndTags ${TESTS_DIR}/X31-DuplicatedTestCases.cpp)
target_link_libraries(DuplicatedTestCases-SameNameAndTags PRIVATE Catch2::Catch2WithMain)
add_test(

59
tests/ExtraTests/X29-CustomArgumentsForReporters.cpp Normal file
View File

@ -0,0 +1,59 @@
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE_1_0.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
/**\file
* Test that custom options are properly passed down to the reporter.
*
* We print out the arguments sorted by key, to have a stable expected
* output.
*/
#include <catch2/catch_test_macros.hpp>
#include <catch2/reporters/catch_reporter_streaming_base.hpp>
#include <catch2/reporters/catch_reporter_registrars.hpp>
#include <algorithm>
#include <iostream>
#include <string>
#include <utility>
#include <vector>
class TestReporter : public Catch::StreamingReporterBase {
public:
TestReporter( Catch::ReporterConfig const& _config ):
StreamingReporterBase( _config ) {
std::cout << "X29 - TestReporter constructed\n";
}
static std::string getDescription() {
return "X29 test reporter";
}
void testRunStarting( Catch::TestRunInfo const& ) override {
std::vector<std::pair<std::string, std::string>> options;
for ( auto const& kv : m_customOptions ) {
options.push_back( kv );
}
std::sort( options.begin(), options.end() );
bool first = true;
for ( auto const& kv : options ) {
if ( !first ) { std::cout << "::"; }
std::cout << kv.first << "=" << kv.second;
first = false;
}
std::cout << '\n';
}
~TestReporter() override;
};
TestReporter::~TestReporter() = default;
CATCH_REGISTER_REPORTER( "test-reporter", TestReporter )
TEST_CASE( "Just a test case to run things" ) {}

40
tests/SelfTest/Baselines/compact.sw.approved.txt
View File

@ -1316,10 +1316,8 @@ CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> }
==
{ console, <default-output> }
CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false
@ -1333,37 +1331,21 @@ CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } }
==
{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } }
==
{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } }
==
{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } }
==
{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}

40
tests/SelfTest/Baselines/compact.sw.multi.approved.txt
View File

@ -1314,10 +1314,8 @@ CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications.empty() for: true
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> }
==
{ console, <default-output> }
CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs().size() == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false for: false == false
@ -1331,37 +1329,21 @@ CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } }
==
{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } }
==
{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } }
==
{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } }
==
{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}

2
tests/SelfTest/Baselines/console.std.approved.txt
View File

@ -1396,5 +1396,5 @@ due to unexpected exception with message:
===============================================================================
test cases: 390 | 314 passed | 69 failed | 7 failed as expected
assertions: 2225 | 2070 passed | 128 failed | 27 failed as expected
assertions: 2223 | 2068 passed | 128 failed | 27 failed as expected

76
tests/SelfTest/Baselines/console.sw.approved.txt
View File

@ -9433,16 +9433,14 @@ with expansion:
!false
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles().size() == 1 )
CHECK( cfg.getReporterSpecs().size() == 1 )
with expansion:
1 == 1
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } )
CHECK( cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } )
with expansion:
{ console, <default-output> }
==
{ console, <default-output> }
{?} == {?}
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9544,11 +9542,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } )
with expansion:
{ { console, <default-output> } }
==
{ { console, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9568,11 +9564,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } )
with expansion:
{ { xml, <default-output> } }
==
{ { xml, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9592,11 +9586,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } )
with expansion:
{ { junit, <default-output> } }
==
{ { junit, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9635,11 +9627,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } )
with expansion:
{ { console, out.txt } }
==
{ { console, out.txt } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9659,34 +9649,12 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } )
with expansion:
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Output file cannot be empty
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("empty filename") )
with expansion:
"Reporter 'console' has empty filename specified as its output. Supply a
filename or remove the colons to use the default output." contains: "empty
filename"
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
@ -9697,16 +9665,14 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) )
CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } )
with expansion:
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
{ {?}, {?} } == { {?}, {?} }
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9718,16 +9684,14 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) )
CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } )
with expansion:
{ { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
{ {?}, {?} } == { {?}, {?} }
-------------------------------------------------------------------------------
Process can be configured on command line
@ -17953,5 +17917,5 @@ Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 390 | 300 passed | 83 failed | 7 failed as expected
assertions: 2240 | 2070 passed | 143 failed | 27 failed as expected
assertions: 2238 | 2068 passed | 143 failed | 27 failed as expected

76
tests/SelfTest/Baselines/console.sw.multi.approved.txt
View File

@ -9431,16 +9431,14 @@ with expansion:
!false
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles().size() == 1 )
CHECK( cfg.getReporterSpecs().size() == 1 )
with expansion:
1 == 1
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } )
CHECK( cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } )
with expansion:
{ console, <default-output> }
==
{ console, <default-output> }
{?} == {?}
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9542,11 +9540,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } )
with expansion:
{ { console, <default-output> } }
==
{ { console, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9566,11 +9562,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } )
with expansion:
{ { xml, <default-output> } }
==
{ { xml, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9590,11 +9584,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } )
with expansion:
{ { junit, <default-output> } }
==
{ { junit, <default-output> } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9633,11 +9625,9 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } )
with expansion:
{ { console, out.txt } }
==
{ { console, out.txt } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
@ -9657,34 +9647,12 @@ with message:
result.errorMessage() := ""
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } )
with expansion:
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
{ {?} } == { {?} }
with message:
result.errorMessage() := ""
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
Output file cannot be empty
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( !result )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("empty filename") )
with expansion:
"Reporter 'console' has empty filename specified as its output. Supply a
filename or remove the colons to use the default output." contains: "empty
filename"
-------------------------------------------------------------------------------
Process can be configured on command line
reporter
@ -9695,16 +9663,14 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) )
CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } )
with expansion:
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
{ {?}, {?} } == { {?}, {?} }
-------------------------------------------------------------------------------
Process can be configured on command line
@ -9716,16 +9682,14 @@ CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
CHECK( cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) )
CHECK( cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) )
with expansion:
{?}
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } )
REQUIRE( config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } )
with expansion:
{ { xml, output.xml }, { console, <default-output> } }
==
{ { xml, output.xml }, { console, <default-output> } }
{ {?}, {?} } == { {?}, {?} }
-------------------------------------------------------------------------------
Process can be configured on command line
@ -17945,5 +17909,5 @@ Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 390 | 300 passed | 83 failed | 7 failed as expected
assertions: 2240 | 2070 passed | 143 failed | 27 failed as expected
assertions: 2238 | 2068 passed | 143 failed | 27 failed as expected

3
tests/SelfTest/Baselines/junit.sw.approved.txt
View File

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact
>
<testsuite name="<exe-name>" errors="17" failures="126" tests="2240" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="126" tests="2238" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@ -1109,7 +1109,6 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Output file cannot be empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>

3
tests/SelfTest/Baselines/junit.sw.multi.approved.txt
View File

@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="<exe-name>" errors="17" failures="126" tests="2240" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="126" tests="2238" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@ -1108,7 +1108,6 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Output file cannot be empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>

1
tests/SelfTest/Baselines/sonarqube.sw.approved.txt
View File

@ -66,7 +66,6 @@
<testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Output file cannot be empty" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/>

1
tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt
View File

@ -65,7 +65,6 @@
<testCase name="Process can be configured on command line/reporter/must match one of the available ones" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Output file cannot be empty" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/All with output files" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" duration="{duration}"/>
<testCase name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" duration="{duration}"/>

28
tests/SelfTest/Baselines/tap.sw.approved.txt
View File

@ -2479,9 +2479,9 @@ ok {test-number} - config.reporterSpecifications.empty() for: true
# Process can be configured on command line
ok {test-number} - !(cfg.hasTestFilters()) for: !false
# Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1
ok {test-number} - cfg.getReporterSpecs().size() == 1 for: 1 == 1
# Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } == { console, <default-output> }
ok {test-number} - cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
# Process can be configured on command line
ok {test-number} - result for: {?}
# Process can be configured on command line
@ -2509,15 +2509,15 @@ ok {test-number} - cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for:
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } == { { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } == { { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
@ -2525,23 +2525,19 @@ ok {test-number} - result.errorMessage(), ContainsSubstring("Unrecognized report
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } == { { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } == { { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } } == { { xml, output.xml }, { junit, output-junit.xml } }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } } == { { xml, output.xml }, { console, <default-output> } }
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
@ -4482,5 +4478,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2240
1..2238

28
tests/SelfTest/Baselines/tap.sw.multi.approved.txt
View File

@ -2477,9 +2477,9 @@ ok {test-number} - config.reporterSpecifications.empty() for: true
# Process can be configured on command line
ok {test-number} - !(cfg.hasTestFilters()) for: !false
# Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles().size() == 1 for: 1 == 1
ok {test-number} - cfg.getReporterSpecs().size() == 1 for: 1 == 1
# Process can be configured on command line
ok {test-number} - cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } for: { console, <default-output> } == { console, <default-output> }
ok {test-number} - cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} } for: {?} == {?}
# Process can be configured on command line
ok {test-number} - result for: {?}
# Process can be configured on command line
@ -2507,15 +2507,15 @@ ok {test-number} - cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for:
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } } == { { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } } == { { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } } == { { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
@ -2523,23 +2523,19 @@ ok {test-number} - result.errorMessage(), ContainsSubstring("Unrecognized report
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } } == { { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - result for: {?} with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } } == { { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } for: { {?} } == { {?} } with 1 message: 'result.errorMessage() := ""'
# Process can be configured on command line
ok {test-number} - !result for: true
ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }) for: {?}
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
ok {test-number} - cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } } == { { xml, output.xml }, { junit, output-junit.xml } }
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } } == { { xml, output.xml }, { console, <default-output> } }
ok {test-number} - config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } } for: { {?}, {?} } == { {?}, {?} }
# Process can be configured on command line
ok {test-number} - !result for: true
# Process can be configured on command line
@ -4474,5 +4470,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2240
1..2238

78
tests/SelfTest/Baselines/xml.sw.approved.txt
View File

@ -11468,7 +11468,7 @@ C
</Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cfg.getReportersAndOutputFiles().size() == 1
cfg.getReporterSpecs().size() == 1
</Original>
<Expanded>
1 == 1
@ -11476,12 +11476,10 @@ C
</Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} }
cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} }
</Original>
<Expanded>
{ console, &lt;default-output> }
==
{ console, &lt;default-output> }
{?} == {?}
</Expanded>
</Expression>
<OverallResults successes="9" failures="0" expectedFailures="0"/>
@ -11618,12 +11616,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} }
config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } }
</Original>
<Expanded>
{ { console, &lt;default-output> } }
==
{ { console, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11648,12 +11644,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} }
config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } }
</Original>
<Expanded>
{ { xml, &lt;default-output> } }
==
{ { xml, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11678,12 +11672,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} }
config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } }
</Original>
<Expanded>
{ { junit, &lt;default-output> } }
==
{ { junit, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11730,12 +11722,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} }
config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } }
</Original>
<Expanded>
{ { console, out.txt } }
==
{ { console, out.txt } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11760,34 +11750,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} }
config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } }
</Original>
<Expanded>
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Output file cannot be empty" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("empty filename")
</Original>
<Expanded>
"Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11799,7 +11765,7 @@ C
<Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" })
cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" })
</Original>
<Expanded>
{?}
@ -11807,12 +11773,10 @@ C
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} }
config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } }
</Original>
<Expanded>
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
{ {?}, {?} } == { {?}, {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11826,7 +11790,7 @@ C
<Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" })
cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" })
</Original>
<Expanded>
{?}
@ -11834,12 +11798,10 @@ C
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} }
config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } }
</Original>
<Expanded>
{ { xml, output.xml }, { console, &lt;default-output> } }
==
{ { xml, output.xml }, { console, &lt;default-output> } }
{ {?}, {?} } == { {?}, {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -21072,6 +21034,6 @@ loose text artifact
</Section>
<OverallResult success="true"/>
</TestCase>
<OverallResults successes="2070" failures="143" expectedFailures="27"/>
<OverallResults successes="2068" failures="143" expectedFailures="27"/>
<OverallResultsCases successes="300" failures="83" expectedFailures="7"/>
</Catch2TestRun>

78
tests/SelfTest/Baselines/xml.sw.multi.approved.txt
View File

@ -11468,7 +11468,7 @@ C
</Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cfg.getReportersAndOutputFiles().size() == 1
cfg.getReporterSpecs().size() == 1
</Original>
<Expanded>
1 == 1
@ -11476,12 +11476,10 @@ C
</Expression>
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cfg.getReportersAndOutputFiles()[0] == Catch::ConfigData::ReporterAndFile{ expectedReporter, {} }
cfg.getReporterSpecs()[0] == Catch::ReporterSpec{ expectedReporter, {}, {}, {} }
</Original>
<Expanded>
{ console, &lt;default-output> }
==
{ console, &lt;default-output> }
{?} == {?}
</Expanded>
</Expression>
<OverallResults successes="9" failures="0" expectedFailures="0"/>
@ -11618,12 +11616,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} }
config.reporterSpecifications == vec_Specs{ { "console", {}, {}, {} } }
</Original>
<Expanded>
{ { console, &lt;default-output> } }
==
{ { console, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11648,12 +11644,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} }
config.reporterSpecifications == vec_Specs{ { "xml", {}, {}, {} } }
</Original>
<Expanded>
{ { xml, &lt;default-output> } }
==
{ { xml, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11678,12 +11672,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} }
config.reporterSpecifications == vec_Specs{ { "junit", {}, {}, {} } }
</Original>
<Expanded>
{ { junit, &lt;default-output> } }
==
{ { junit, &lt;default-output> } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11730,12 +11722,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} }
config.reporterSpecifications == vec_Specs{ { "console", "out.txt"s, {}, {} } }
</Original>
<Expanded>
{ { console, out.txt } }
==
{ { console, out.txt } }
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11760,34 +11750,10 @@ C
</Info>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} }
config.reporterSpecifications == vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } }
</Original>
<Expanded>
{ { console, C:\Temp\out.txt } }
==
{ { console, C:\Temp\out.txt } }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
<Section name="reporter" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Output file cannot be empty" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!result
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
result.errorMessage(), ContainsSubstring("empty filename")
</Original>
<Expanded>
"Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
{ {?} } == { {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11799,7 +11765,7 @@ C
<Section name="All with output files" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" })
cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" })
</Original>
<Expanded>
{?}
@ -11807,12 +11773,10 @@ C
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} }
config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "junit", "output-junit.xml"s, {}, {} } }
</Original>
<Expanded>
{ { xml, output.xml }, { junit, output-junit.xml } }
==
{ { xml, output.xml }, { junit, output-junit.xml } }
{ {?}, {?} } == { {?}, {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -11826,7 +11790,7 @@ C
<Section name="Mixed output files and default output" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" })
cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" })
</Original>
<Expanded>
{?}
@ -11834,12 +11798,10 @@ C
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} }
config.reporterSpecifications == vec_Specs{ { "xml", "output.xml"s, {}, {} }, { "console", {}, {}, {} } }
</Original>
<Expanded>
{ { xml, output.xml }, { console, &lt;default-output> } }
==
{ { xml, output.xml }, { console, &lt;default-output> } }
{ {?}, {?} } == { {?}, {?} }
</Expanded>
</Expression>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
@ -21071,6 +21033,6 @@ There is no extra whitespace here
</Section>
<OverallResult success="true"/>
</TestCase>
<OverallResults successes="2070" failures="143" expectedFailures="27"/>
<OverallResults successes="2068" failures="143" expectedFailures="27"/>
<OverallResultsCases successes="300" failures="83" expectedFailures="7"/>
</Catch2TestRun>

47
tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp
View File

@ -361,9 +361,9 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
#endif
;
CHECK( cfg.getReportersAndOutputFiles().size() == 1 );
CHECK( cfg.getReportersAndOutputFiles()[0] ==
Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } );
CHECK( cfg.getReporterSpecs().size() == 1 );
CHECK( cfg.getReporterSpecs()[0] ==
Catch::ReporterSpec{ expectedReporter, {}, {}, {} } );
}
SECTION("test lists") {
@ -399,28 +399,31 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
}
SECTION("reporter") {
using vec_ReporterAndFile = std::vector<Catch::ConfigData::ReporterAndFile>;
using vec_Specs = std::vector<Catch::ReporterSpec>;
using namespace std::string_literals;
SECTION("-r/console") {
auto result = cli.parse({"test", "-r", "console"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} });
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "console", {}, {}, {} } } );
}
SECTION("-r/xml") {
auto result = cli.parse({"test", "-r", "xml"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} });
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", {}, {}, {} } } );
}
SECTION("--reporter/junit") {
auto result = cli.parse({"test", "--reporter", "junit"});
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} });
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "junit", {}, {}, {} } } );
}
SECTION("must match one of the available ones") {
auto result = cli.parse({"test", "--reporter", "unsupported"});
@ -429,34 +432,34 @@ TEST_CASE( "Process can be configured on command line", "[config][command-line]"
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Unrecognized reporter"));
}
SECTION("With output file") {
auto result = cli.parse({ "test", "-r", "console::out.txt" });
auto result = cli.parse({ "test", "-r", "console::out=out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} });
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "console", "out.txt"s, {}, {} } } );
}
SECTION("With Windows-like absolute path as output file") {
auto result = cli.parse({ "test", "-r", "console::C:\\Temp\\out.txt" });
auto result = cli.parse({ "test", "-r", "console::out=C:\\Temp\\out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} });
}
SECTION("Output file cannot be empty") {
auto result = cli.parse({"test", "--reporter", "console::"});
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("empty filename"));
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "console", "C:\\Temp\\out.txt"s, {}, {} } } );
}
SECTION("Multiple reporters") {
SECTION("All with output files") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} });
CHECK(cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "junit::out=output-junit.xml" }));
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", "output.xml"s, {}, {} },
{ "junit", "output-junit.xml"s, {}, {} } } );
}
SECTION("Mixed output files and default output") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} });
CHECK(cli.parse({ "test", "-r", "xml::out=output.xml", "-r", "console" }));
REQUIRE( config.reporterSpecifications ==
vec_Specs{ { "xml", "output.xml"s, {}, {} },
{ "console", {}, {}, {} } } );
}
SECTION("cannot have multiple reporters with default output") {
auto result = cli.parse({ "test", "-r", "console", "-r", "xml::output.xml", "-r", "junit" });
auto result = cli.parse({ "test", "-r", "console", "-r", "xml::out=output.xml", "-r", "junit" });
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file."));
}

9
tests/SelfTest/IntrospectiveTests/Reporters.tests.cpp
View File

@ -89,7 +89,8 @@ TEST_CASE( "Reporter's write listings to provided stream", "[reporters]" ) {
Catch::ConfigData config_data;
Catch::Config config( config_data );
Catch::ReporterConfig rep_config( &config, &sstream );
Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
auto reporter = factory.second->create( rep_config );
DYNAMIC_SECTION( factory.first << " reporter lists tags" ) {
@ -176,7 +177,8 @@ TEST_CASE("Multireporter calls reporters and listeners in correct order",
Catch::ConfigData config_data;
Catch::Config config( config_data );
StringIStream sstream;
Catch::ReporterConfig rep_config( &config, &sstream );
Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
// We add reporters before listeners, to check that internally they
// get sorted properly, and listeners are called first anyway.
@ -229,7 +231,8 @@ TEST_CASE("Multireporter updates ReporterPreferences properly",
Catch::ConfigData config_data;
Catch::Config config( config_data );
StringIStream sstream;
Catch::ReporterConfig rep_config( &config, &sstream );
Catch::ReporterConfig rep_config(
&config, &sstream, Catch::ColourMode::None, {} );
Catch::MultiReporter multiReporter( &config );
// Post init defaults

2
tools/scripts/approvalTests.py
View File

@ -220,7 +220,7 @@ common_args = ["~[!nonportable]~[!benchmark]~[approvals] *", "-s", "-w", "NoAsse
filenames = ['{}.sw.multi'.format(reporter) for reporter in reporters]
reporter_args = []
for reporter, filename in zip(reporters, filenames):
reporter_args += ['-r', '{}::{}'.format(reporter, get_rawResultsPath(filename))]
reporter_args += ['-r', '{}::out={}'.format(reporter, get_rawResultsPath(filename))]
run_test("default.sw.multi", common_args + reporter_args)
check_outputs("default.sw.multi")