3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-22 13:26:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add new SKIP macro for skipping tests at runtime (#2360)

Browse Source 
* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This commit is contained in:
Philip Salzmann 2023-01-12 15:01:47 +01:00 committed by GitHub
parent 52066dbc2a
commit d548be26e3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
47 changed files with 3722 additions and 2171 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/Readme.md
View File

@ -11,6 +11,7 @@ Once you're up and running consider the following reference material.
* [Logging macros](logging.md#top) * [Logging macros](logging.md#top)
* [Test cases and sections](test-cases-and-sections.md#top) * [Test cases and sections](test-cases-and-sections.md#top)
* [Test fixtures](test-fixtures.md#top) * [Test fixtures](test-fixtures.md#top)
* [Skipping tests at runtime](skipping.md#top)
* [Reporters (output customization)](reporters.md#top) * [Reporters (output customization)](reporters.md#top)
* [Event Listeners](event-listeners.md#top) * [Event Listeners](event-listeners.md#top)
* [Data Generators (value parameterized tests)](generators.md#top) * [Data Generators (value parameterized tests)](generators.md#top)

8
docs/command-line.md
View File

@ -561,10 +561,10 @@ processes, as is done with the [Bazel test sharding](https://docs.bazel.build/ve
> Introduced in Catch2 3.0.1. > Introduced in Catch2 3.0.1.
By default, Catch2 test binaries return non-0 exit code if no tests were By default, Catch2 test binaries return non-0 exit code if no tests were run,
run, e.g. if the binary was compiled with no tests, or the provided test e.g. if the binary was compiled with no tests, the provided test spec matched no
spec matched no tests. This flag overrides that, so a test run with no tests, or all tests [were skipped at runtime](skipping.md#top). This flag
tests still returns 0. overrides that, so a test run with no tests still returns 0.
## Output verbosity ## Output verbosity
``` ```

9
docs/deprecations.md
View File

@ -26,6 +26,15 @@ to accurately probe the environment for this information so the flag
where it will export `BAZEL_TEST=1` for purposes like the above. Catch2 where it will export `BAZEL_TEST=1` for purposes like the above. Catch2
will now instead inspect the environment instead of relying on build configuration. will now instead inspect the environment instead of relying on build configuration.
### `IEventLister::skipTest( TestCaseInfo const& testInfo )`
This event (including implementations in derived classes such as `ReporterBase`)
is deprecated and will be removed in the next major release. It is currently
invoked for all test cases that are not going to be executed due to the test run
being aborted (when using `--abort` or `--abortx`). It is however
**NOT** invoked for test cases that are [explicitly skipped using the `SKIP`
macro](skipping.md#top).
--- ---
[Home](Readme.md#top) [Home](Readme.md#top)

72
docs/skipping.md Normal file
View File

@ -0,0 +1,72 @@
<a id="top"></a>
# Skipping Test Cases at Runtime
> [Introduced](https://github.com/catchorg/Catch2/pull/2360) in Catch2 X.Y.Z.
In some situations it may not be possible to meaningfully execute a test case, for example when the system under test is missing certain hardware capabilities.
If the required conditions can only be determined at runtime, it often doesn't make sense to consider such a test case as either passed or failed, because it simply can not run at all.
To properly express such scenarios, Catch2 allows to explicitly _skip_ test cases, using the `SKIP` macro:
**SKIP(** _message expression_ **)**
Example usage:
```c++
TEST_CASE("copy files between drives") {
if(getNumberOfHardDrives() < 2) {
SKIP("at least two hard drives required");
}
// ...
}
```
This test case is then reported as _skipped_ instead of _passed_ or _failed_.
The `SKIP` macro behaves similarly to an explicit [`FAIL`](logging.md#top), in that it is the last expression that will be executed:
```c++
TEST_CASE("my test") {
printf("foo");
SKIP();
printf("bar"); // not printed
}
```
However a failed assertion _before_ a `SKIP` still causes the entire test case to fail:
```c++
TEST_CASE("failing test") {
CHECK(1 == 2);
SKIP();
}
```
## Interaction with Sections and Generators
Sections, nested sections as well as test cases with [generators](generators.md#top) can all be individually skipped, with the rest executing as usual:
```c++
TEST_CASE("complex test case") {
int value = GENERATE(2, 4, 6);
SECTION("a") {
SECTION("a1") { CHECK(value < 8); }
SECTION("a2") {
if (value == 4) {
SKIP();
}
CHECK(value % 2 == 0);
}
}
}
```
This test case will report 5 passing assertions; one for each of the three values in section `a1`, as well as one for each in `a2`, except for when `value == 4`.
Note that as soon as one section is skipped, the entire test case will be reported as _skipped_ (unless there is a failing assertion, in which case it will be reported as _failed_ instead).
If all test cases in a run are skipped, Catch2 returns a non-zero exit code by default.
This can be overridden using the [--allow-running-no-tests](command-line.md#no-tests-override) flag.
---
[Home](Readme.md#top)

6
src/catch2/catch_session.cpp
View File

@ -341,6 +341,12 @@ namespace Catch {
return 2; return 2;
} }
if ( totals.testCases.total() > 0 &&
totals.testCases.total() == totals.testCases.skipped
&& !m_config->zeroTestsCountAsSuccess() ) {
return 4;
}
// Note that on unices only the lower 8 bits are usually used, clamping // Note that on unices only the lower 8 bits are usually used, clamping
// the return value to 255 prevents false negative when some multiple // the return value to 255 prevents false negative when some multiple
// of 256 tests has failed // of 256 tests has failed

4
src/catch2/catch_test_macros.hpp
View File

@ -49,6 +49,7 @@
#define CATCH_FAIL( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ ) #define CATCH_FAIL( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
#define CATCH_FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ ) #define CATCH_FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
#define CATCH_SUCCEED( ... ) INTERNAL_CATCH_MSG( "CATCH_SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ ) #define CATCH_SUCCEED( ... ) INTERNAL_CATCH_MSG( "CATCH_SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
#define CATCH_SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
#if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE) #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@ -102,6 +103,7 @@
#define CATCH_FAIL( ... ) (void)(0) #define CATCH_FAIL( ... ) (void)(0)
#define CATCH_FAIL_CHECK( ... ) (void)(0) #define CATCH_FAIL_CHECK( ... ) (void)(0)
#define CATCH_SUCCEED( ... ) (void)(0) #define CATCH_SUCCEED( ... ) (void)(0)
#define CATCH_SKIP( ... ) (void)(0)
#define CATCH_STATIC_REQUIRE( ... ) (void)(0) #define CATCH_STATIC_REQUIRE( ... ) (void)(0)
#define CATCH_STATIC_REQUIRE_FALSE( ... ) (void)(0) #define CATCH_STATIC_REQUIRE_FALSE( ... ) (void)(0)
@ -146,6 +148,7 @@
#define FAIL( ... ) INTERNAL_CATCH_MSG( "FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ ) #define FAIL( ... ) INTERNAL_CATCH_MSG( "FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
#define FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ ) #define FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
#define SUCCEED( ... ) INTERNAL_CATCH_MSG( "SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ ) #define SUCCEED( ... ) INTERNAL_CATCH_MSG( "SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
#define SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
#if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE) #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@ -198,6 +201,7 @@
#define FAIL( ... ) (void)(0) #define FAIL( ... ) (void)(0)
#define FAIL_CHECK( ... ) (void)(0) #define FAIL_CHECK( ... ) (void)(0)
#define SUCCEED( ... ) (void)(0) #define SUCCEED( ... ) (void)(0)
#define SKIP( ... ) (void)(0)
#define STATIC_REQUIRE( ... ) (void)(0) #define STATIC_REQUIRE( ... ) (void)(0)
#define STATIC_REQUIRE_FALSE( ... ) (void)(0) #define STATIC_REQUIRE_FALSE( ... ) (void)(0)

8
src/catch2/catch_totals.cpp
View File

@ -14,6 +14,7 @@ namespace Catch {
diff.passed = passed - other.passed; diff.passed = passed - other.passed;
diff.failed = failed - other.failed; diff.failed = failed - other.failed;
diff.failedButOk = failedButOk - other.failedButOk; diff.failedButOk = failedButOk - other.failedButOk;
diff.skipped = skipped - other.skipped;
return diff; return diff;
} }
@ -21,14 +22,15 @@ namespace Catch {
passed += other.passed; passed += other.passed;
failed += other.failed; failed += other.failed;
failedButOk += other.failedButOk; failedButOk += other.failedButOk;
skipped += other.skipped;
return *this; return *this;
} }
std::uint64_t Counts::total() const { std::uint64_t Counts::total() const {
return passed + failed + failedButOk; return passed + failed + failedButOk + skipped;
} }
bool Counts::allPassed() const { bool Counts::allPassed() const {
return failed == 0 && failedButOk == 0; return failed == 0 && failedButOk == 0 && skipped == 0;
} }
bool Counts::allOk() const { bool Counts::allOk() const {
return failed == 0; return failed == 0;
@ -53,6 +55,8 @@ namespace Catch {
++diff.testCases.failed; ++diff.testCases.failed;
else if( diff.assertions.failedButOk > 0 ) else if( diff.assertions.failedButOk > 0 )
++diff.testCases.failedButOk; ++diff.testCases.failedButOk;
else if ( diff.assertions.skipped > 0 )
++ diff.testCases.skipped;
else else
++diff.testCases.passed; ++diff.testCases.passed;
return diff; return diff;

1
src/catch2/catch_totals.hpp
View File

@ -23,6 +23,7 @@ namespace Catch {
std::uint64_t passed = 0; std::uint64_t passed = 0;
std::uint64_t failed = 0; std::uint64_t failed = 0;
std::uint64_t failedButOk = 0; std::uint64_t failedButOk = 0;
std::uint64_t skipped = 0;
}; };
struct Totals { struct Totals {

7
src/catch2/interfaces/catch_interfaces_reporter.hpp
View File

@ -242,7 +242,12 @@ namespace Catch {
*/ */
virtual void testRunEnded( TestRunStats const& testRunStats ) = 0; virtual void testRunEnded( TestRunStats const& testRunStats ) = 0;
//! Called with test cases that are skipped due to the test run aborting /**
* Called with test cases that are skipped due to the test run aborting.
* NOT called for test cases that are explicitly skipped using the `SKIP` macro.
*
* Deprecated - will be removed in the next major release.
*/
virtual void skipTest( TestCaseInfo const& testInfo ) = 0; virtual void skipTest( TestCaseInfo const& testInfo ) = 0;
//! Called if a fatal error (signal/structured exception) occured //! Called if a fatal error (signal/structured exception) occured

7
src/catch2/internal/catch_assertion_handler.cpp
View File

@ -50,6 +50,13 @@ namespace Catch {
if (m_reaction.shouldThrow) { if (m_reaction.shouldThrow) {
throw_test_failure_exception(); throw_test_failure_exception();
} }
if ( m_reaction.shouldSkip ) {
#if !defined( CATCH_CONFIG_DISABLE_EXCEPTIONS )
throw Catch::TestSkipException();
#else
CATCH_ERROR( "Explicitly skipping tests during runtime requires exceptions" );
#endif
}
} }
void AssertionHandler::setCompleted() { void AssertionHandler::setCompleted() {
m_completed = true; m_completed = true;

1
src/catch2/internal/catch_assertion_handler.hpp
View File

@ -22,6 +22,7 @@ namespace Catch {
struct AssertionReaction { struct AssertionReaction {
bool shouldDebugBreak = false; bool shouldDebugBreak = false;
bool shouldThrow = false; bool shouldThrow = false;
bool shouldSkip = false;
}; };
class AssertionHandler { class AssertionHandler {

1
src/catch2/internal/catch_console_colour.hpp
View File

@ -47,6 +47,7 @@ namespace Catch {
Error = BrightRed, Error = BrightRed,
Success = Green, Success = Green,
Skip = LightGrey,
OriginalExpression = Cyan, OriginalExpression = Cyan,
ReconstructedExpression = BrightYellow, ReconstructedExpression = BrightYellow,

3
src/catch2/internal/catch_exception_translator_registry.cpp
View File

@ -44,6 +44,9 @@ namespace Catch {
catch( TestFailureException& ) { catch( TestFailureException& ) {
std::rethrow_exception(std::current_exception()); std::rethrow_exception(std::current_exception());
} }
catch( TestSkipException& ) {
std::rethrow_exception(std::current_exception());
}
catch( std::exception const& ex ) { catch( std::exception const& ex ) {
return ex.what(); return ex.what();
} }

2
src/catch2/internal/catch_result_type.hpp
View File

@ -16,6 +16,8 @@ namespace Catch {
Ok = 0, Ok = 0,
Info = 1, Info = 1,
Warning = 2, Warning = 2,
// TODO: Should explicit skip be considered "not OK" (cf. isOk)? I.e., should it have the failure bit?
ExplicitSkip = 4,
FailureBit = 0x10, FailureBit = 0x10,

12
src/catch2/internal/catch_run_context.cpp
View File

@ -270,6 +270,9 @@ namespace Catch {
if (result.getResultType() == ResultWas::Ok) { if (result.getResultType() == ResultWas::Ok) {
m_totals.assertions.passed++; m_totals.assertions.passed++;
m_lastAssertionPassed = true; m_lastAssertionPassed = true;
} else if (result.getResultType() == ResultWas::ExplicitSkip) {
m_totals.assertions.skipped++;
m_lastAssertionPassed = true;
} else if (!result.succeeded()) { } else if (!result.succeeded()) {
m_lastAssertionPassed = false; m_lastAssertionPassed = false;
if (result.isOk()) { if (result.isOk()) {
@ -475,6 +478,8 @@ namespace Catch {
duration = timer.getElapsedSeconds(); duration = timer.getElapsedSeconds();
} CATCH_CATCH_ANON (TestFailureException&) { } CATCH_CATCH_ANON (TestFailureException&) {
// This just means the test was aborted due to failure // This just means the test was aborted due to failure
} CATCH_CATCH_ANON (TestSkipException&) {
// This just means the test was explicitly skipped
} CATCH_CATCH_ALL { } CATCH_CATCH_ALL {
// Under CATCH_CONFIG_FAST_COMPILE, unexpected exceptions under REQUIRE assertions // Under CATCH_CONFIG_FAST_COMPILE, unexpected exceptions under REQUIRE assertions
// are reported without translation at the point of origin. // are reported without translation at the point of origin.
@ -571,8 +576,13 @@ namespace Catch {
data.message = static_cast<std::string>(message); data.message = static_cast<std::string>(message);
AssertionResult assertionResult{ m_lastAssertionInfo, data }; AssertionResult assertionResult{ m_lastAssertionInfo, data };
assertionEnded( assertionResult ); assertionEnded( assertionResult );
if( !assertionResult.isOk() ) if ( !assertionResult.isOk() ) {
populateReaction( reaction ); populateReaction( reaction );
} else if ( resultType == ResultWas::ExplicitSkip ) {
// TODO: Need to handle this explicitly, as ExplicitSkip is
// considered "OK"
reaction.shouldSkip = true;
}
} }
void RunContext::handleUnexpectedExceptionNotThrown( void RunContext::handleUnexpectedExceptionNotThrown(
AssertionInfo const& info, AssertionInfo const& info,

3
src/catch2/internal/catch_test_failure_exception.hpp
View File

@ -20,6 +20,9 @@ namespace Catch {
*/ */
[[noreturn]] void throw_test_failure_exception(); [[noreturn]] void throw_test_failure_exception();
//! Used to signal that the remainder of a test should be skipped
struct TestSkipException{};
} // namespace Catch } // namespace Catch
#endif // CATCH_TEST_FAILURE_EXCEPTION_HPP_INCLUDED #endif // CATCH_TEST_FAILURE_EXCEPTION_HPP_INCLUDED

4
src/catch2/reporters/catch_reporter_automake.cpp
View File

@ -17,7 +17,9 @@ namespace Catch {
void AutomakeReporter::testCaseEnded(TestCaseStats const& _testCaseStats) { void AutomakeReporter::testCaseEnded(TestCaseStats const& _testCaseStats) {
// Possible values to emit are PASS, XFAIL, SKIP, FAIL, XPASS and ERROR. // Possible values to emit are PASS, XFAIL, SKIP, FAIL, XPASS and ERROR.
m_stream << ":test-result: "; m_stream << ":test-result: ";
if (_testCaseStats.totals.assertions.allPassed()) { if ( _testCaseStats.totals.testCases.skipped > 0 ) {
m_stream << "SKIP";
} else if (_testCaseStats.totals.assertions.allPassed()) {
m_stream << "PASS"; m_stream << "PASS";
} else if (_testCaseStats.totals.assertions.allOk()) { } else if (_testCaseStats.totals.assertions.allOk()) {
m_stream << "XFAIL"; m_stream << "XFAIL";

7
src/catch2/reporters/catch_reporter_compact.cpp
View File

@ -105,6 +105,11 @@ public:
printIssue("explicitly"); printIssue("explicitly");
printRemainingMessages(Colour::None); printRemainingMessages(Colour::None);
break; break;
case ResultWas::ExplicitSkip:
printResultType(Colour::Skip, "skipped"_sr);
printMessage();
printRemainingMessages();
break;
// These cases are here to prevent compiler warnings // These cases are here to prevent compiler warnings
case ResultWas::Unknown: case ResultWas::Unknown:
case ResultWas::FailureBit: case ResultWas::FailureBit:
@ -220,7 +225,7 @@ private:
// Drop out if result was successful and we're not printing those // Drop out if result was successful and we're not printing those
if( !m_config->includeSuccessfulResults() && result.isOk() ) { if( !m_config->includeSuccessfulResults() && result.isOk() ) {
if( result.getResultType() != ResultWas::Warning ) if( result.getResultType() != ResultWas::Warning && result.getResultType() != ResultWas::ExplicitSkip )
return; return;
printInfoMessages = false; printInfoMessages = false;
} }

31
src/catch2/reporters/catch_reporter_console.cpp
View File

@ -111,6 +111,14 @@ public:
if (_stats.infoMessages.size() > 1) if (_stats.infoMessages.size() > 1)
messageLabel = "explicitly with messages"; messageLabel = "explicitly with messages";
break; break;
case ResultWas::ExplicitSkip:
colour = Colour::Skip;
passOrFail = "SKIPPED"_sr;
if (_stats.infoMessages.size() == 1)
messageLabel = "explicitly with message";
if (_stats.infoMessages.size() > 1)
messageLabel = "explicitly with messages";
break;
// These cases are here to prevent compiler warnings // These cases are here to prevent compiler warnings
case ResultWas::Unknown: case ResultWas::Unknown:
case ResultWas::FailureBit: case ResultWas::FailureBit:
@ -185,13 +193,16 @@ std::size_t makeRatio( std::uint64_t number, std::uint64_t total ) {
return (ratio == 0 && number > 0) ? 1 : static_cast<std::size_t>(ratio); return (ratio == 0 && number > 0) ? 1 : static_cast<std::size_t>(ratio);
} }
std::size_t& findMax( std::size_t& i, std::size_t& j, std::size_t& k ) { std::size_t&
if (i > j && i > k) findMax( std::size_t& i, std::size_t& j, std::size_t& k, std::size_t& l ) {
if (i > j && i > k && i > l)
return i; return i;
else if (j > k) else if (j > k && j > l)
return j; return j;
else else if (k > l)
return k; return k;
else
return l;
} }
enum class Justification { Left, Right }; enum class Justification { Left, Right };
@ -400,7 +411,8 @@ void ConsoleReporter::assertionEnded(AssertionStats const& _assertionStats) {
bool includeResults = m_config->includeSuccessfulResults() || !result.isOk(); bool includeResults = m_config->includeSuccessfulResults() || !result.isOk();
// Drop out if result was successful but we're not printing them. // Drop out if result was successful but we're not printing them.
if (!includeResults && result.getResultType() != ResultWas::Warning) // TODO: Make configurable whether skips should be printed
if (!includeResults && result.getResultType() != ResultWas::Warning && result.getResultType() != ResultWas::ExplicitSkip)
return; return;
lazyPrint(); lazyPrint();
@ -603,10 +615,11 @@ void ConsoleReporter::printTotalsDivider(Totals const& totals) {
std::size_t failedRatio = makeRatio(totals.testCases.failed, totals.testCases.total()); std::size_t failedRatio = makeRatio(totals.testCases.failed, totals.testCases.total());
std::size_t failedButOkRatio = makeRatio(totals.testCases.failedButOk, totals.testCases.total()); std::size_t failedButOkRatio = makeRatio(totals.testCases.failedButOk, totals.testCases.total());
std::size_t passedRatio = makeRatio(totals.testCases.passed, totals.testCases.total()); std::size_t passedRatio = makeRatio(totals.testCases.passed, totals.testCases.total());
while (failedRatio + failedButOkRatio + passedRatio < CATCH_CONFIG_CONSOLE_WIDTH - 1) std::size_t skippedRatio = makeRatio(totals.testCases.skipped, totals.testCases.total());
findMax(failedRatio, failedButOkRatio, passedRatio)++; while (failedRatio + failedButOkRatio + passedRatio + skippedRatio < CATCH_CONFIG_CONSOLE_WIDTH - 1)
findMax(failedRatio, failedButOkRatio, passedRatio, skippedRatio)++;
while (failedRatio + failedButOkRatio + passedRatio > CATCH_CONFIG_CONSOLE_WIDTH - 1) while (failedRatio + failedButOkRatio + passedRatio > CATCH_CONFIG_CONSOLE_WIDTH - 1)
findMax(failedRatio, failedButOkRatio, passedRatio)--; findMax(failedRatio, failedButOkRatio, passedRatio, skippedRatio)--;
m_stream << m_colour->guardColour( Colour::Error ) m_stream << m_colour->guardColour( Colour::Error )
<< std::string( failedRatio, '=' ) << std::string( failedRatio, '=' )
@ -619,6 +632,8 @@ void ConsoleReporter::printTotalsDivider(Totals const& totals) {
m_stream << m_colour->guardColour( Colour::Success ) m_stream << m_colour->guardColour( Colour::Success )
<< std::string( passedRatio, '=' ); << std::string( passedRatio, '=' );
} }
m_stream << m_colour->guardColour( Colour::Skip )
<< std::string( skippedRatio, '=' );
} else { } else {
m_stream << m_colour->guardColour( Colour::Warning ) m_stream << m_colour->guardColour( Colour::Warning )
<< std::string( CATCH_CONFIG_CONSOLE_WIDTH - 1, '=' ); << std::string( CATCH_CONFIG_CONSOLE_WIDTH - 1, '=' );

9
src/catch2/reporters/catch_reporter_helpers.cpp
View File

@ -316,15 +316,22 @@ namespace Catch {
} }
std::vector<SummaryColumn> columns; std::vector<SummaryColumn> columns;
// Don't include "skipped assertions" in total count
const auto totalAssertionCount =
totals.assertions.total() - totals.assertions.skipped;
columns.push_back( SummaryColumn( "", Colour::None ) columns.push_back( SummaryColumn( "", Colour::None )
.addRow( totals.testCases.total() ) .addRow( totals.testCases.total() )
.addRow( totals.assertions.total() ) ); .addRow( totalAssertionCount ) );
columns.push_back( SummaryColumn( "passed", Colour::Success ) columns.push_back( SummaryColumn( "passed", Colour::Success )
.addRow( totals.testCases.passed ) .addRow( totals.testCases.passed )
.addRow( totals.assertions.passed ) ); .addRow( totals.assertions.passed ) );
columns.push_back( SummaryColumn( "failed", Colour::ResultError ) columns.push_back( SummaryColumn( "failed", Colour::ResultError )
.addRow( totals.testCases.failed ) .addRow( totals.testCases.failed )
.addRow( totals.assertions.failed ) ); .addRow( totals.assertions.failed ) );
columns.push_back( SummaryColumn( "skipped", Colour::Skip )
.addRow( totals.testCases.skipped )
// Don't print "skipped assertions"
.addRow( 0 ) );
columns.push_back( columns.push_back(
SummaryColumn( "failed as expected", Colour::ResultExpectedFailure ) SummaryColumn( "failed as expected", Colour::ResultExpectedFailure )
.addRow( totals.testCases.failedButOk ) .addRow( totals.testCases.failedButOk )

14
src/catch2/reporters/catch_reporter_junit.cpp
View File

@ -132,6 +132,7 @@ namespace Catch {
xml.writeAttribute( "name"_sr, stats.runInfo.name ); xml.writeAttribute( "name"_sr, stats.runInfo.name );
xml.writeAttribute( "errors"_sr, unexpectedExceptions ); xml.writeAttribute( "errors"_sr, unexpectedExceptions );
xml.writeAttribute( "failures"_sr, stats.totals.assertions.failed-unexpectedExceptions ); xml.writeAttribute( "failures"_sr, stats.totals.assertions.failed-unexpectedExceptions );
xml.writeAttribute( "skipped"_sr, stats.totals.assertions.skipped );
xml.writeAttribute( "tests"_sr, stats.totals.assertions.total() ); xml.writeAttribute( "tests"_sr, stats.totals.assertions.total() );
xml.writeAttribute( "hostname"_sr, "tbd"_sr ); // !TBD xml.writeAttribute( "hostname"_sr, "tbd"_sr ); // !TBD
if( m_config->showDurations() == ShowDurations::Never ) if( m_config->showDurations() == ShowDurations::Never )
@ -244,7 +245,8 @@ namespace Catch {
void JunitReporter::writeAssertion( AssertionStats const& stats ) { void JunitReporter::writeAssertion( AssertionStats const& stats ) {
AssertionResult const& result = stats.assertionResult; AssertionResult const& result = stats.assertionResult;
if( !result.isOk() ) { if ( !result.isOk() ||
result.getResultType() == ResultWas::ExplicitSkip ) {
std::string elementName; std::string elementName;
switch( result.getResultType() ) { switch( result.getResultType() ) {
case ResultWas::ThrewException: case ResultWas::ThrewException:
@ -256,7 +258,9 @@ namespace Catch {
case ResultWas::DidntThrowException: case ResultWas::DidntThrowException:
elementName = "failure"; elementName = "failure";
break; break;
case ResultWas::ExplicitSkip:
elementName = "skipped";
break;
// We should never see these here: // We should never see these here:
case ResultWas::Info: case ResultWas::Info:
case ResultWas::Warning: case ResultWas::Warning:
@ -274,7 +278,9 @@ namespace Catch {
xml.writeAttribute( "type"_sr, result.getTestMacroName() ); xml.writeAttribute( "type"_sr, result.getTestMacroName() );
ReusableStringStream rss; ReusableStringStream rss;
if (stats.totals.assertions.total() > 0) { if ( result.getResultType() == ResultWas::ExplicitSkip ) {
rss << "SKIPPED\n";
} else {
rss << "FAILED" << ":\n"; rss << "FAILED" << ":\n";
if (result.hasExpression()) { if (result.hasExpression()) {
rss << " "; rss << " ";
@ -285,8 +291,6 @@ namespace Catch {
rss << "with expansion:\n"; rss << "with expansion:\n";
rss << TextFlow::Column(result.getExpandedExpression()).indent(2) << '\n'; rss << TextFlow::Column(result.getExpandedExpression()).indent(2) << '\n';
} }
} else {
rss << '\n';
} }
if( !result.getMessage().empty() ) if( !result.getMessage().empty() )

15
src/catch2/reporters/catch_reporter_sonarqube.cpp
View File

@ -97,7 +97,8 @@ namespace Catch {
void SonarQubeReporter::writeAssertion(AssertionStats const& stats, bool okToFail) { void SonarQubeReporter::writeAssertion(AssertionStats const& stats, bool okToFail) {
AssertionResult const& result = stats.assertionResult; AssertionResult const& result = stats.assertionResult;
if (!result.isOk()) { if ( !result.isOk() ||
result.getResultType() == ResultWas::ExplicitSkip ) {
std::string elementName; std::string elementName;
if (okToFail) { if (okToFail) {
elementName = "skipped"; elementName = "skipped";
@ -108,15 +109,13 @@ namespace Catch {
elementName = "error"; elementName = "error";
break; break;
case ResultWas::ExplicitFailure: case ResultWas::ExplicitFailure:
elementName = "failure";
break;
case ResultWas::ExpressionFailed: case ResultWas::ExpressionFailed:
elementName = "failure";
break;
case ResultWas::DidntThrowException: case ResultWas::DidntThrowException:
elementName = "failure"; elementName = "failure";
break; break;
case ResultWas::ExplicitSkip:
elementName = "skipped";
break;
// We should never see these here: // We should never see these here:
case ResultWas::Info: case ResultWas::Info:
case ResultWas::Warning: case ResultWas::Warning:
@ -136,7 +135,9 @@ namespace Catch {
xml.writeAttribute("message"_sr, messageRss.str()); xml.writeAttribute("message"_sr, messageRss.str());
ReusableStringStream textRss; ReusableStringStream textRss;
if (stats.totals.assertions.total() > 0) { if ( result.getResultType() == ResultWas::ExplicitSkip ) {
textRss << "SKIPPED\n";
} else {
textRss << "FAILED:\n"; textRss << "FAILED:\n";
if (result.hasExpression()) { if (result.hasExpression()) {
textRss << '\t' << result.getExpressionInMacro() << '\n'; textRss << '\t' << result.getExpressionInMacro() << '\n';

6
src/catch2/reporters/catch_reporter_tap.cpp
View File

@ -100,6 +100,12 @@ namespace Catch {
printIssue("explicitly"_sr); printIssue("explicitly"_sr);
printRemainingMessages(Colour::None); printRemainingMessages(Colour::None);
break; break;
case ResultWas::ExplicitSkip:
printResultType(tapPassedString);
printIssue(" # SKIP"_sr);
printMessage();
printRemainingMessages();
break;
// These cases are here to prevent compiler warnings // These cases are here to prevent compiler warnings
case ResultWas::Unknown: case ResultWas::Unknown:
case ResultWas::FailureBit: case ResultWas::FailureBit:

22
src/catch2/reporters/catch_reporter_teamcity.cpp
View File

@ -59,7 +59,8 @@ namespace Catch {
void TeamCityReporter::assertionEnded(AssertionStats const& assertionStats) { void TeamCityReporter::assertionEnded(AssertionStats const& assertionStats) {
AssertionResult const& result = assertionStats.assertionResult; AssertionResult const& result = assertionStats.assertionResult;
if (!result.isOk()) { if ( !result.isOk() ||
result.getResultType() == ResultWas::ExplicitSkip ) {
ReusableStringStream msg; ReusableStringStream msg;
if (!m_headerPrintedForThisSection) if (!m_headerPrintedForThisSection)
@ -84,6 +85,9 @@ namespace Catch {
case ResultWas::ExplicitFailure: case ResultWas::ExplicitFailure:
msg << "explicit failure"; msg << "explicit failure";
break; break;
case ResultWas::ExplicitSkip:
msg << "explicit skip";
break;
// We shouldn't get here because of the isOk() test // We shouldn't get here because of the isOk() test
case ResultWas::Ok: case ResultWas::Ok:
@ -111,18 +115,16 @@ namespace Catch {
" " << result.getExpandedExpression() << '\n'; " " << result.getExpandedExpression() << '\n';
} }
if (currentTestCaseInfo->okToFail()) { if ( result.getResultType() == ResultWas::ExplicitSkip ) {
m_stream << "##teamcity[testIgnored";
} else if ( currentTestCaseInfo->okToFail() ) {
msg << "- failure ignore as test marked as 'ok to fail'\n"; msg << "- failure ignore as test marked as 'ok to fail'\n";
m_stream << "##teamcity[testIgnored" m_stream << "##teamcity[testIgnored";
<< " name='" << escape(currentTestCaseInfo->name) << '\''
<< " message='" << escape(msg.str()) << '\''
<< "]\n";
} else { } else {
m_stream << "##teamcity[testFailed" m_stream << "##teamcity[testFailed";
<< " name='" << escape(currentTestCaseInfo->name) << '\''
<< " message='" << escape(msg.str()) << '\''
<< "]\n";
} }
m_stream << " name='" << escape( currentTestCaseInfo->name ) << '\''
<< " message='" << escape( msg.str() ) << '\'' << "]\n";
} }
m_stream.flush(); m_stream.flush();
} }

23
src/catch2/reporters/catch_reporter_xml.cpp
View File

@ -108,9 +108,10 @@ namespace Catch {
} }
// Drop out if result was successful but we're not printing them. // Drop out if result was successful but we're not printing them.
if( !includeResults && result.getResultType() != ResultWas::Warning ) if ( !includeResults && result.getResultType() != ResultWas::Warning &&
result.getResultType() != ResultWas::ExplicitSkip ) {
return; return;
}
// Print the expression if there is one. // Print the expression if there is one.
if( result.hasExpression() ) { if( result.hasExpression() ) {
@ -153,6 +154,12 @@ namespace Catch {
m_xml.writeText( result.getMessage() ); m_xml.writeText( result.getMessage() );
m_xml.endElement(); m_xml.endElement();
break; break;
case ResultWas::ExplicitSkip:
m_xml.startElement( "Skip" );
writeSourceInfo( result.getSourceInfo() );
m_xml.writeText( result.getMessage() );
m_xml.endElement();
break;
default: default:
break; break;
} }
@ -164,14 +171,17 @@ namespace Catch {
void XmlReporter::sectionEnded( SectionStats const& sectionStats ) { void XmlReporter::sectionEnded( SectionStats const& sectionStats ) {
StreamingReporterBase::sectionEnded( sectionStats ); StreamingReporterBase::sectionEnded( sectionStats );
if ( --m_sectionDepth > 0 ) { if ( --m_sectionDepth > 0 ) {
{
XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResults" ); XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResults" );
e.writeAttribute( "successes"_sr, sectionStats.assertions.passed ); e.writeAttribute( "successes"_sr, sectionStats.assertions.passed );
e.writeAttribute( "failures"_sr, sectionStats.assertions.failed ); e.writeAttribute( "failures"_sr, sectionStats.assertions.failed );
e.writeAttribute( "expectedFailures"_sr, sectionStats.assertions.failedButOk ); e.writeAttribute( "expectedFailures"_sr, sectionStats.assertions.failedButOk );
e.writeAttribute( "skipped"_sr, sectionStats.assertions.skipped > 0 );
if ( m_config->showDurations() == ShowDurations::Always ) if ( m_config->showDurations() == ShowDurations::Always )
e.writeAttribute( "durationInSeconds"_sr, sectionStats.durationInSeconds ); e.writeAttribute( "durationInSeconds"_sr, sectionStats.durationInSeconds );
}
// Ends assertion tag
m_xml.endElement(); m_xml.endElement();
} }
} }
@ -180,6 +190,7 @@ namespace Catch {
StreamingReporterBase::testCaseEnded( testCaseStats ); StreamingReporterBase::testCaseEnded( testCaseStats );
XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResult" ); XmlWriter::ScopedElement e = m_xml.scopedElement( "OverallResult" );
e.writeAttribute( "success"_sr, testCaseStats.totals.assertions.allOk() ); e.writeAttribute( "success"_sr, testCaseStats.totals.assertions.allOk() );
e.writeAttribute( "skips"_sr, testCaseStats.totals.assertions.skipped );
if ( m_config->showDurations() == ShowDurations::Always ) if ( m_config->showDurations() == ShowDurations::Always )
e.writeAttribute( "durationInSeconds"_sr, m_testCaseTimer.getElapsedSeconds() ); e.writeAttribute( "durationInSeconds"_sr, m_testCaseTimer.getElapsedSeconds() );
@ -197,11 +208,13 @@ namespace Catch {
m_xml.scopedElement( "OverallResults" ) m_xml.scopedElement( "OverallResults" )
.writeAttribute( "successes"_sr, testRunStats.totals.assertions.passed ) .writeAttribute( "successes"_sr, testRunStats.totals.assertions.passed )
.writeAttribute( "failures"_sr, testRunStats.totals.assertions.failed ) .writeAttribute( "failures"_sr, testRunStats.totals.assertions.failed )
.writeAttribute( "expectedFailures"_sr, testRunStats.totals.assertions.failedButOk ); .writeAttribute( "expectedFailures"_sr, testRunStats.totals.assertions.failedButOk )
.writeAttribute( "skips"_sr, testRunStats.totals.assertions.skipped );
m_xml.scopedElement( "OverallResultsCases") m_xml.scopedElement( "OverallResultsCases")
.writeAttribute( "successes"_sr, testRunStats.totals.testCases.passed ) .writeAttribute( "successes"_sr, testRunStats.totals.testCases.passed )
.writeAttribute( "failures"_sr, testRunStats.totals.testCases.failed ) .writeAttribute( "failures"_sr, testRunStats.totals.testCases.failed )
.writeAttribute( "expectedFailures"_sr, testRunStats.totals.testCases.failedButOk ); .writeAttribute( "expectedFailures"_sr, testRunStats.totals.testCases.failedButOk )
.writeAttribute( "skips"_sr, testRunStats.totals.testCases.skipped );
m_xml.endElement(); m_xml.endElement();
} }

5
tests/CMakeLists.txt
View File

@ -116,6 +116,7 @@ set(TEST_SOURCES
${SELF_TEST_DIR}/UsageTests/Generators.tests.cpp ${SELF_TEST_DIR}/UsageTests/Generators.tests.cpp
${SELF_TEST_DIR}/UsageTests/Message.tests.cpp ${SELF_TEST_DIR}/UsageTests/Message.tests.cpp
${SELF_TEST_DIR}/UsageTests/Misc.tests.cpp ${SELF_TEST_DIR}/UsageTests/Misc.tests.cpp
${SELF_TEST_DIR}/UsageTests/Skip.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringByte.tests.cpp ${SELF_TEST_DIR}/UsageTests/ToStringByte.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringChrono.tests.cpp ${SELF_TEST_DIR}/UsageTests/ToStringChrono.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringGeneral.tests.cpp ${SELF_TEST_DIR}/UsageTests/ToStringGeneral.tests.cpp
@ -272,6 +273,10 @@ add_test(NAME TestSpecs::OverrideFailureWithNoMatchedTests
COMMAND $<TARGET_FILE:SelfTest> "___nonexistent_test___" --allow-running-no-tests COMMAND $<TARGET_FILE:SelfTest> "___nonexistent_test___" --allow-running-no-tests
) )
add_test(NAME TestSpecs::OverrideAllSkipFailure
COMMAND $<TARGET_FILE:SelfTest> "tests can be skipped dynamically at runtime" --allow-running-no-tests
)
add_test(NAME TestSpecs::NonMatchingTestSpecIsRoundTrippable add_test(NAME TestSpecs::NonMatchingTestSpecIsRoundTrippable
COMMAND $<TARGET_FILE:SelfTest> Tracker, "this test does not exist" "[nor does this tag]" COMMAND $<TARGET_FILE:SelfTest> Tracker, "this test does not exist" "[nor does this tag]"
) )

17
tests/ExtraTests/CMakeLists.txt
View File

@ -488,15 +488,32 @@ set_tests_properties(TestSpecs::EmptySpecWithNoTestsFails
PROPERTIES PROPERTIES
WILL_FAIL ON WILL_FAIL ON
) )
add_test( add_test(
NAME TestSpecs::OverrideFailureWithEmptySpec NAME TestSpecs::OverrideFailureWithEmptySpec
COMMAND $<TARGET_FILE:NoTests> --allow-running-no-tests COMMAND $<TARGET_FILE:NoTests> --allow-running-no-tests
) )
add_test( add_test(
NAME List::Listeners::WorksWithoutRegisteredListeners NAME List::Listeners::WorksWithoutRegisteredListeners
COMMAND $<TARGET_FILE:NoTests> --list-listeners COMMAND $<TARGET_FILE:NoTests> --list-listeners
) )
add_executable(AllSkipped ${TESTS_DIR}/X93-AllSkipped.cpp)
target_link_libraries(AllSkipped PRIVATE Catch2::Catch2WithMain)
add_test(
NAME TestSpecs::SkippingAllTestsFails
COMMAND $<TARGET_FILE:AllSkipped>
)
set_tests_properties(TestSpecs::SkippingAllTestsFails
PROPERTIES
WILL_FAIL ON
)
set( EXTRA_TEST_BINARIES set( EXTRA_TEST_BINARIES
AllSkipped
PrefixedMacros PrefixedMacros
DisabledMacros DisabledMacros
DisabledExceptions-DefaultHandler DisabledExceptions-DefaultHandler

16
tests/ExtraTests/X93-AllSkipped.cpp Normal file
View File

@ -0,0 +1,16 @@
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
#include <catch2/catch_test_macros.hpp>
TEST_CASE( "this test case is being skipped" ) { SKIP(); }
TEST_CASE( "all sections in this test case are being skipped" ) {
SECTION( "A" ) { SKIP(); }
SECTION( "B" ) { SKIP(); }
}

12
tests/SelfTest/Baselines/automake.sw.approved.txt
View File

@ -297,6 +297,7 @@ Message from section two
:test-result: PASS X/level/1/b :test-result: PASS X/level/1/b
:test-result: PASS XmlEncode :test-result: PASS XmlEncode
:test-result: PASS XmlWriter writes boolean attributes as true/false :test-result: PASS XmlWriter writes boolean attributes as true/false
:test-result: SKIP a succeeding test can still be skipped
:test-result: PASS analyse no analysis :test-result: PASS analyse no analysis
:test-result: PASS array<int, N> -> toString :test-result: PASS array<int, N> -> toString
:test-result: PASS benchmark function call :test-result: PASS benchmark function call
@ -309,10 +310,14 @@ Message from section two
:test-result: PASS comparisons between const int variables :test-result: PASS comparisons between const int variables
:test-result: PASS comparisons between int variables :test-result: PASS comparisons between int variables
:test-result: PASS convertToBits :test-result: PASS convertToBits
:test-result: SKIP dynamic skipping works with generators
:test-result: PASS empty tags are not allowed :test-result: PASS empty tags are not allowed
:test-result: PASS erfc_inv :test-result: PASS erfc_inv
:test-result: PASS estimate_clock_resolution :test-result: PASS estimate_clock_resolution
:test-result: PASS even more nested SECTION tests :test-result: PASS even more nested SECTION tests
:test-result: XFAIL failed assertions before SKIP cause test case to fail
:test-result: XFAIL failing for some generator values causes entire test case to fail
:test-result: XFAIL failing in some unskipped sections causes entire test case to fail
:test-result: FAIL first tag :test-result: FAIL first tag
loose text artifact loose text artifact
:test-result: FAIL has printf :test-result: FAIL has printf
@ -331,6 +336,10 @@ loose text artifact
:test-result: FAIL mix info, unscoped info and warning :test-result: FAIL mix info, unscoped info and warning
:test-result: FAIL more nested SECTION tests :test-result: FAIL more nested SECTION tests
:test-result: PASS nested SECTION tests :test-result: PASS nested SECTION tests
a!
b1!
!
:test-result: FAIL nested sections can be skipped dynamically at runtime
:test-result: PASS non streamable - with conv. op :test-result: PASS non streamable - with conv. op
:test-result: PASS non-copyable objects :test-result: PASS non-copyable objects
:test-result: PASS normal_cdf :test-result: PASS normal_cdf
@ -352,9 +361,11 @@ loose text artifact
:test-result: PASS run_for_at_least, chronometer :test-result: PASS run_for_at_least, chronometer
:test-result: PASS run_for_at_least, int :test-result: PASS run_for_at_least, int
:test-result: FAIL second tag :test-result: FAIL second tag
:test-result: SKIP sections can be skipped dynamically at runtime
:test-result: FAIL send a single char to INFO :test-result: FAIL send a single char to INFO
:test-result: FAIL sends information to INFO :test-result: FAIL sends information to INFO
:test-result: PASS shortened hide tags are split apart :test-result: PASS shortened hide tags are split apart
:test-result: SKIP skipped tests can optionally provide a reason
:test-result: PASS splitString :test-result: PASS splitString
:test-result: FAIL stacks unscoped info in loops :test-result: FAIL stacks unscoped info in loops
:test-result: PASS startsWith :test-result: PASS startsWith
@ -376,6 +387,7 @@ loose text artifact
:test-result: PASS strlen3 :test-result: PASS strlen3
:test-result: PASS tables :test-result: PASS tables
:test-result: PASS tags with dots in later positions are not parsed as hidden :test-result: PASS tags with dots in later positions are not parsed as hidden
:test-result: SKIP tests can be skipped dynamically at runtime
:test-result: FAIL thrown std::strings are translated :test-result: FAIL thrown std::strings are translated
:test-result: PASS toString on const wchar_t const pointer returns the string contents :test-result: PASS toString on const wchar_t const pointer returns the string contents
:test-result: PASS toString on const wchar_t pointer returns the string contents :test-result: PASS toString on const wchar_t pointer returns the string contents

9
tests/SelfTest/Baselines/automake.sw.multi.approved.txt
View File

@ -290,6 +290,7 @@
:test-result: PASS X/level/1/b :test-result: PASS X/level/1/b
:test-result: PASS XmlEncode :test-result: PASS XmlEncode
:test-result: PASS XmlWriter writes boolean attributes as true/false :test-result: PASS XmlWriter writes boolean attributes as true/false
:test-result: SKIP a succeeding test can still be skipped
:test-result: PASS analyse no analysis :test-result: PASS analyse no analysis
:test-result: PASS array<int, N> -> toString :test-result: PASS array<int, N> -> toString
:test-result: PASS benchmark function call :test-result: PASS benchmark function call
@ -302,10 +303,14 @@
:test-result: PASS comparisons between const int variables :test-result: PASS comparisons between const int variables
:test-result: PASS comparisons between int variables :test-result: PASS comparisons between int variables
:test-result: PASS convertToBits :test-result: PASS convertToBits
:test-result: SKIP dynamic skipping works with generators
:test-result: PASS empty tags are not allowed :test-result: PASS empty tags are not allowed
:test-result: PASS erfc_inv :test-result: PASS erfc_inv
:test-result: PASS estimate_clock_resolution :test-result: PASS estimate_clock_resolution
:test-result: PASS even more nested SECTION tests :test-result: PASS even more nested SECTION tests
:test-result: XFAIL failed assertions before SKIP cause test case to fail
:test-result: XFAIL failing for some generator values causes entire test case to fail
:test-result: XFAIL failing in some unskipped sections causes entire test case to fail
:test-result: FAIL first tag :test-result: FAIL first tag
:test-result: FAIL has printf :test-result: FAIL has printf
:test-result: PASS is_unary_function :test-result: PASS is_unary_function
@ -323,6 +328,7 @@
:test-result: FAIL mix info, unscoped info and warning :test-result: FAIL mix info, unscoped info and warning
:test-result: FAIL more nested SECTION tests :test-result: FAIL more nested SECTION tests
:test-result: PASS nested SECTION tests :test-result: PASS nested SECTION tests
:test-result: FAIL nested sections can be skipped dynamically at runtime
:test-result: PASS non streamable - with conv. op :test-result: PASS non streamable - with conv. op
:test-result: PASS non-copyable objects :test-result: PASS non-copyable objects
:test-result: PASS normal_cdf :test-result: PASS normal_cdf
@ -344,9 +350,11 @@
:test-result: PASS run_for_at_least, chronometer :test-result: PASS run_for_at_least, chronometer
:test-result: PASS run_for_at_least, int :test-result: PASS run_for_at_least, int
:test-result: FAIL second tag :test-result: FAIL second tag
:test-result: SKIP sections can be skipped dynamically at runtime
:test-result: FAIL send a single char to INFO :test-result: FAIL send a single char to INFO
:test-result: FAIL sends information to INFO :test-result: FAIL sends information to INFO
:test-result: PASS shortened hide tags are split apart :test-result: PASS shortened hide tags are split apart
:test-result: SKIP skipped tests can optionally provide a reason
:test-result: PASS splitString :test-result: PASS splitString
:test-result: FAIL stacks unscoped info in loops :test-result: FAIL stacks unscoped info in loops
:test-result: PASS startsWith :test-result: PASS startsWith
@ -368,6 +376,7 @@
:test-result: PASS strlen3 :test-result: PASS strlen3
:test-result: PASS tables :test-result: PASS tables
:test-result: PASS tags with dots in later positions are not parsed as hidden :test-result: PASS tags with dots in later positions are not parsed as hidden
:test-result: SKIP tests can be skipped dynamically at runtime
:test-result: FAIL thrown std::strings are translated :test-result: FAIL thrown std::strings are translated
:test-result: PASS toString on const wchar_t const pointer returns the string contents :test-result: PASS toString on const wchar_t const pointer returns the string contents
:test-result: PASS toString on const wchar_t pointer returns the string contents :test-result: PASS toString on const wchar_t pointer returns the string contents

26
tests/SelfTest/Baselines/compact.sw.approved.txt
View File

@ -2060,6 +2060,8 @@ Xml.tests.cpp:<line number>: passed: encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F
Xml.tests.cpp:<line number>: passed: stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> Xml.tests.cpp:<line number>: passed: stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?>
<Element1 attr1="true" attr2="false"/> <Element1 attr1="true" attr2="false"/>
" ( contains: "attr1="true"" and contains: "attr2="false"" ) " ( contains: "attr1="true"" and contains: "attr2="false"" )
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.point.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.point.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.lower_bound.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.lower_bound.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.upper_bound.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.upper_bound.count() == 23 for: 23.0 == 23
@ -2149,6 +2151,9 @@ FloatingPoint.tests.cpp:<line number>: passed: convertToBits( -0. ) == ( 1ULL <<
9223372036854775808 (0x<hex digits>) 9223372036854775808 (0x<hex digits>)
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1 FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1 FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 41'
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
Tag.tests.cpp:<line number>: passed: Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo) Tag.tests.cpp:<line number>: passed: Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo)
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.103560) == Approx(-0.09203687623843015) for: -0.0920368762 == Approx( -0.0920368762 ) InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.103560) == Approx(-0.09203687623843015) for: -0.0920368762 == Approx( -0.0920368762 )
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.067400) == Approx(-0.05980291115763361) for: -0.0598029112 == Approx( -0.0598029112 ) InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.067400) == Approx(-0.05980291115763361) for: -0.0598029112 == Approx( -0.0598029112 )
@ -2158,6 +2163,14 @@ InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: failed: 3 == 4
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
loose text artifact loose text artifact
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value' Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value'
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary2)>::value' Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary2)>::value'
@ -2215,6 +2228,10 @@ Misc.tests.cpp:<line number>: passed: a < b for: 1 < 2
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2 Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
Misc.tests.cpp:<line number>: passed: b != a for: 2 != 1 Misc.tests.cpp:<line number>: passed: b != a for: 2 != 1
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2 Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
a!
b1!
Skip.tests.cpp:<line number>: skipped:
!
Tricky.tests.cpp:<line number>: passed: s == "7" for: "7" == "7" Tricky.tests.cpp:<line number>: passed: s == "7" for: "7" == "7"
Tricky.tests.cpp:<line number>: passed: ti == typeid(int) for: {?} == {?} Tricky.tests.cpp:<line number>: passed: ti == typeid(int) for: {?} == {?}
InternalBenchmark.tests.cpp:<line number>: passed: normal_cdf(0.000000) == Approx(0.50000000000000000) for: 0.5 == Approx( 0.5 ) InternalBenchmark.tests.cpp:<line number>: passed: normal_cdf(0.000000) == Approx(0.50000000000000000) for: 0.5 == Approx( 0.5 )
@ -2299,9 +2316,13 @@ InternalBenchmark.tests.cpp:<line number>: passed: x >= old_x for: 128 >= 64
InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed >= time for: 128 ns >= 100 ns InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed >= time for: 128 ns >= 100 ns
InternalBenchmark.tests.cpp:<line number>: passed: Timing.result == Timing.iterations + 17 for: 145 == 145 InternalBenchmark.tests.cpp:<line number>: passed: Timing.result == Timing.iterations + 17 for: 145 == 145
InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations >= time.count() for: 128 >= 100 InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations >= time.count() for: 128 >= 100
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: failed: false with 1 message: '3' Misc.tests.cpp:<line number>: failed: false with 1 message: '3'
Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7' Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7'
Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} ) Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { } StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc", ','), Equals(std::vector<StringRef>{"abc"}) for: { abc } Equals: { abc } StringManip.tests.cpp:<line number>: passed: splitStringRef("abc", ','), Equals(std::vector<StringRef>{"abc"}) for: { abc } Equals: { abc }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc,def", ','), Equals(std::vector<StringRef>{"abc", "def"}) for: { abc, def } Equals: { abc, def } StringManip.tests.cpp:<line number>: passed: splitStringRef("abc,def", ','), Equals(std::vector<StringRef>{"abc", "def"}) for: { abc, def } Equals: { abc, def }
@ -2367,6 +2388,7 @@ Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_
Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)) for: 6 == 6 Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)) for: 6 == 6
Tag.tests.cpp:<line number>: passed: testcase.tags.size() == 1 for: 1 == 1 Tag.tests.cpp:<line number>: passed: testcase.tags.size() == 1 for: 1 == 1
Tag.tests.cpp:<line number>: passed: testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag Tag.tests.cpp:<line number>: passed: testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag
Skip.tests.cpp:<line number>: skipped:
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'Why would you throw a std::string?' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'Why would you throw a std::string?'
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load"" Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load"" Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
@ -2463,7 +2485,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0 InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
test cases: 395 | 305 passed | 83 failed | 7 failed as expected test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected

23
tests/SelfTest/Baselines/compact.sw.multi.approved.txt
View File

@ -2053,6 +2053,8 @@ Xml.tests.cpp:<line number>: passed: encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F
Xml.tests.cpp:<line number>: passed: stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> Xml.tests.cpp:<line number>: passed: stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?>
<Element1 attr1="true" attr2="false"/> <Element1 attr1="true" attr2="false"/>
" ( contains: "attr1="true"" and contains: "attr2="false"" ) " ( contains: "attr1="true"" and contains: "attr2="false"" )
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.point.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.point.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.lower_bound.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.lower_bound.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.upper_bound.count() == 23 for: 23.0 == 23 InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.upper_bound.count() == 23 for: 23.0 == 23
@ -2142,6 +2144,9 @@ FloatingPoint.tests.cpp:<line number>: passed: convertToBits( -0. ) == ( 1ULL <<
9223372036854775808 (0x<hex digits>) 9223372036854775808 (0x<hex digits>)
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1 FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1 FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 41'
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
Tag.tests.cpp:<line number>: passed: Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo) Tag.tests.cpp:<line number>: passed: Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo)
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.103560) == Approx(-0.09203687623843015) for: -0.0920368762 == Approx( -0.0920368762 ) InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.103560) == Approx(-0.09203687623843015) for: -0.0920368762 == Approx( -0.0920368762 )
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.067400) == Approx(-0.05980291115763361) for: -0.0598029112 == Approx( -0.0598029112 ) InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.067400) == Approx(-0.05980291115763361) for: -0.0598029112 == Approx( -0.0598029112 )
@ -2151,6 +2156,14 @@ InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: failed: 3 == 4
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value' Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value'
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary2)>::value' Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary2)>::value'
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary3)>::value' Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary3)>::value'
@ -2207,6 +2220,7 @@ Misc.tests.cpp:<line number>: passed: a < b for: 1 < 2
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2 Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
Misc.tests.cpp:<line number>: passed: b != a for: 2 != 1 Misc.tests.cpp:<line number>: passed: b != a for: 2 != 1
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2 Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
Skip.tests.cpp:<line number>: skipped:
Tricky.tests.cpp:<line number>: passed: s == "7" for: "7" == "7" Tricky.tests.cpp:<line number>: passed: s == "7" for: "7" == "7"
Tricky.tests.cpp:<line number>: passed: ti == typeid(int) for: {?} == {?} Tricky.tests.cpp:<line number>: passed: ti == typeid(int) for: {?} == {?}
InternalBenchmark.tests.cpp:<line number>: passed: normal_cdf(0.000000) == Approx(0.50000000000000000) for: 0.5 == Approx( 0.5 ) InternalBenchmark.tests.cpp:<line number>: passed: normal_cdf(0.000000) == Approx(0.50000000000000000) for: 0.5 == Approx( 0.5 )
@ -2291,9 +2305,13 @@ InternalBenchmark.tests.cpp:<line number>: passed: x >= old_x for: 128 >= 64
InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed >= time for: 128 ns >= 100 ns InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed >= time for: 128 ns >= 100 ns
InternalBenchmark.tests.cpp:<line number>: passed: Timing.result == Timing.iterations + 17 for: 145 == 145 InternalBenchmark.tests.cpp:<line number>: passed: Timing.result == Timing.iterations + 17 for: 145 == 145
InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations >= time.count() for: 128 >= 100 InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations >= time.count() for: 128 >= 100
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: failed: false with 1 message: '3' Misc.tests.cpp:<line number>: failed: false with 1 message: '3'
Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7' Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7'
Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} ) Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { } StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc", ','), Equals(std::vector<StringRef>{"abc"}) for: { abc } Equals: { abc } StringManip.tests.cpp:<line number>: passed: splitStringRef("abc", ','), Equals(std::vector<StringRef>{"abc"}) for: { abc } Equals: { abc }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc,def", ','), Equals(std::vector<StringRef>{"abc", "def"}) for: { abc, def } Equals: { abc, def } StringManip.tests.cpp:<line number>: passed: splitStringRef("abc,def", ','), Equals(std::vector<StringRef>{"abc", "def"}) for: { abc, def } Equals: { abc, def }
@ -2359,6 +2377,7 @@ Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_
Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)) for: 6 == 6 Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)) for: 6 == 6
Tag.tests.cpp:<line number>: passed: testcase.tags.size() == 1 for: 1 == 1 Tag.tests.cpp:<line number>: passed: testcase.tags.size() == 1 for: 1 == 1
Tag.tests.cpp:<line number>: passed: testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag Tag.tests.cpp:<line number>: passed: testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag
Skip.tests.cpp:<line number>: skipped:
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'Why would you throw a std::string?' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'Why would you throw a std::string?'
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load"" Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load"" Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
@ -2455,7 +2474,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0 InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
test cases: 395 | 305 passed | 83 failed | 7 failed as expected test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected

133
tests/SelfTest/Baselines/console.std.approved.txt
View File

@ -1164,6 +1164,14 @@ Exception.tests.cpp:<line number>: FAILED:
due to unexpected exception with message: due to unexpected exception with message:
unexpected exception unexpected exception
-------------------------------------------------------------------------------
a succeeding test can still be skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
checkedElse, failing checkedElse, failing
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -1186,6 +1194,87 @@ Misc.tests.cpp:<line number>: FAILED:
with expansion: with expansion:
false false
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 41
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
-------------------------------------------------------------------------------
failed assertions before SKIP cause test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
CHECK( 3 == 4 )
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
loose text artifact loose text artifact
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
just failure just failure
@ -1304,6 +1393,19 @@ Misc.tests.cpp:<line number>: FAILED:
with expansion: with expansion:
1 == 2 1 == 2
a!
b1!
-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
B
B2
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
!
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
not prints unscoped info from previous failures not prints unscoped info from previous failures
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -1338,6 +1440,15 @@ Message.tests.cpp:<line number>: FAILED:
with message: with message:
this SHOULD be seen only ONCE this SHOULD be seen only ONCE
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
send a single char to INFO send a single char to INFO
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -1361,6 +1472,16 @@ with messages:
hi hi
i := 7 i := 7
-------------------------------------------------------------------------------
skipped tests can optionally provide a reason
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
stacks unscoped info in loops stacks unscoped info in loops
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -1383,6 +1504,14 @@ with messages:
5 5
6 6
-------------------------------------------------------------------------------
tests can be skipped dynamically at runtime
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
thrown std::strings are translated thrown std::strings are translated
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -1394,6 +1523,6 @@ due to unexpected exception with message:
Why would you throw a std::string? Why would you throw a std::string?
=============================================================================== ===============================================================================
test cases: 395 | 319 passed | 69 failed | 7 failed as expected test cases: 404 | 319 passed | 69 failed | 6 skipped | 10 failed as expected
assertions: 2148 | 1993 passed | 128 failed | 27 failed as expected assertions: 2156 | 1997 passed | 128 failed | 31 failed as expected

182
tests/SelfTest/Baselines/console.sw.approved.txt
View File

@ -14659,6 +14659,16 @@ with expansion:
<Element1 attr1="true" attr2="false"/> <Element1 attr1="true" attr2="false"/>
" ( contains: "attr1="true"" and contains: "attr2="false"" ) " ( contains: "attr1="true"" and contains: "attr2="false"" )
-------------------------------------------------------------------------------
a succeeding test can still be skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
analyse no analysis analyse no analysis
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15204,6 +15214,34 @@ FloatingPoint.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
1 == 1 1 == 1
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 41
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
empty tags are not allowed empty tags are not allowed
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15279,6 +15317,67 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
failed assertions before SKIP cause test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
CHECK( 3 == 4 )
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
first tag first tag
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15775,6 +15874,40 @@ Misc.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
1 != 2 1 != 2
a-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
A
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
No assertions in section 'A'
!
b1-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
B
B1
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
No assertions in section 'B1'
!
-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
B
B2
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
!
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
non streamable - with conv. op non streamable - with conv. op
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16376,6 +16509,33 @@ Misc.tests.cpp:<line number>
No assertions in test case 'second tag' No assertions in test case 'second tag'
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
also not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
send a single char to INFO send a single char to INFO
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16410,6 +16570,16 @@ Tag.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
{ {?}, {?} } ( Contains: {?} and Contains: {?} ) { {?}, {?} } ( Contains: {?} and Contains: {?} )
-------------------------------------------------------------------------------
skipped tests can optionally provide a reason
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
splitString splitString
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16837,6 +17007,14 @@ Tag.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
magic.tag == magic.tag magic.tag == magic.tag
-------------------------------------------------------------------------------
tests can be skipped dynamically at runtime
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
thrown std::strings are translated thrown std::strings are translated
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -17544,6 +17722,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 395 | 305 passed | 83 failed | 7 failed as expected test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected

179
tests/SelfTest/Baselines/console.sw.multi.approved.txt
View File

@ -14652,6 +14652,16 @@ with expansion:
<Element1 attr1="true" attr2="false"/> <Element1 attr1="true" attr2="false"/>
" ( contains: "attr1="true"" and contains: "attr2="false"" ) " ( contains: "attr1="true"" and contains: "attr2="false"" )
-------------------------------------------------------------------------------
a succeeding test can still be skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
analyse no analysis analyse no analysis
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15197,6 +15207,34 @@ FloatingPoint.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
1 == 1 1 == 1
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 41
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
dynamic skipping works with generators
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
empty tags are not allowed empty tags are not allowed
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15272,6 +15310,67 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
failed assertions before SKIP cause test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
CHECK( 3 == 4 )
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
-------------------------------------------------------------------------------
failing for some generator values causes entire test case to fail
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
failing in some unskipped sections causes entire test case to fail
not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: FAILED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
first tag first tag
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -15767,6 +15866,37 @@ Misc.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
1 != 2 1 != 2
-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
A
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
No assertions in section 'A'
-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
B
B1
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
No assertions in section 'B1'
-------------------------------------------------------------------------------
nested sections can be skipped dynamically at runtime
B
B2
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
non streamable - with conv. op non streamable - with conv. op
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16368,6 +16498,33 @@ Misc.tests.cpp:<line number>
No assertions in test case 'second tag' No assertions in test case 'second tag'
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
-------------------------------------------------------------------------------
sections can be skipped dynamically at runtime
also not skipped
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: PASSED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
send a single char to INFO send a single char to INFO
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16402,6 +16559,16 @@ Tag.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
{ {?}, {?} } ( Contains: {?} and Contains: {?} ) { {?}, {?} } ( Contains: {?} and Contains: {?} )
-------------------------------------------------------------------------------
skipped tests can optionally provide a reason
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
explicitly with message:
skipping because answer = 43
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
splitString splitString
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -16829,6 +16996,14 @@ Tag.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
magic.tag == magic.tag magic.tag == magic.tag
-------------------------------------------------------------------------------
tests can be skipped dynamically at runtime
-------------------------------------------------------------------------------
Skip.tests.cpp:<line number>
...............................................................................
Skip.tests.cpp:<line number>: SKIPPED:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
thrown std::strings are translated thrown std::strings are translated
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -17536,6 +17711,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 395 | 305 passed | 83 failed | 7 failed as expected test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected

3
tests/SelfTest/Baselines/default.sw.multi.approved.txt
View File

@ -6,3 +6,6 @@ A string sent to stderr via clog
Message from section one Message from section one
Message from section two Message from section two
loose text artifact loose text artifact
a!
b1!
!

100
tests/SelfTest/Baselines/junit.sw.approved.txt
View File

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact <testsuitesloose text artifact
> >
<testsuite name="<exe-name>" errors="17" failures="126" tests="2163" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="128" skipped="11" tests="2184" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties> <properties>
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/> <property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
@ -1591,6 +1591,12 @@ at Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (1)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlEncode/string with control char (1)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (x7F)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlEncode/string with control char (x7F)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlWriter writes boolean attributes as true/false" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlWriter writes boolean attributes as true/false" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="a succeeding test can still be skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="analyse no analysis" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="analyse no analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="array&lt;int, N> -> toString" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="array&lt;int, N> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="benchmark function call/without chronometer" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="benchmark function call/without chronometer" time="{duration}" status="run"/>
@ -1625,12 +1631,67 @@ at Misc.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="comparisons between const int variables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="comparisons between const int variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="comparisons between int variables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="comparisons between int variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="convertToBits" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="convertToBits" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="dynamic skipping works with generators" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 41
at Skip.tests.cpp:<line number>
</skipped>
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="empty tags are not allowed" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="empty tags are not allowed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="erfc_inv" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="erfc_inv" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="estimate_clock_resolution" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="estimate_clock_resolution" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/d (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/d (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/e (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/e (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/f (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/f (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="failed assertions before SKIP cause test case to fail" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure message="3 == 4" type="CHECK">
FAILED:
CHECK( 3 == 4 )
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing for some generator values causes entire test case to fail" time="{duration}" status="run">
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/not skipped" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="is_unary_function" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="is_unary_function" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="just failure" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="just failure" time="{duration}" status="run">
<failure type="FAIL"> <failure type="FAIL">
@ -1743,6 +1804,19 @@ at Misc.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/less than" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/less than" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B2/B" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B" time="{duration}" status="run">
<system-out>
a!
b1!
!
</system-out>
</testcase>
<testcase classname="<exe-name>.global" name="non streamable - with conv. op" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="non streamable - with conv. op" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="non-copyable objects" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="non-copyable objects" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="normal_cdf" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="normal_cdf" time="{duration}" status="run"/>
@ -1794,6 +1868,14 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="resolution" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="resolution" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, chronometer" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="run_for_at_least, chronometer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, int" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="run_for_at_least, int" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/not skipped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/also not skipped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="send a single char to INFO" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="send a single char to INFO" time="{duration}" status="run">
<failure message="false" type="REQUIRE"> <failure message="false" type="REQUIRE">
FAILED: FAILED:
@ -1812,6 +1894,13 @@ at Message.tests.cpp:<line number>
</failure> </failure>
</testcase> </testcase>
<testcase classname="<exe-name>.global" name="shortened hide tags are split apart" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="shortened hide tags are split apart" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="skipped tests can optionally provide a reason" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="splitString" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="splitString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stacks unscoped info in loops" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="stacks unscoped info in loops" time="{duration}" status="run">
<failure message="false" type="CHECK"> <failure message="false" type="CHECK">
@ -1856,6 +1945,12 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="strlen3" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="strlen3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="tables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tags with dots in later positions are not parsed as hidden" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="tags with dots in later positions are not parsed as hidden" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tests can be skipped dynamically at runtime" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="thrown std::strings are translated" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="thrown std::strings are translated" time="{duration}" status="run">
<error type="TEST_CASE"> <error type="TEST_CASE">
FAILED: FAILED:
@ -1905,6 +2000,9 @@ This would not be caught previously
A string sent directly to stdout A string sent directly to stdout
Message from section one Message from section one
Message from section two Message from section two
a!
b1!
!
</system-out> </system-out>
<system-err> <system-err>
Nor would this Nor would this

100
tests/SelfTest/Baselines/junit.sw.multi.approved.txt
View File

@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<testsuites> <testsuites>
<testsuite name="<exe-name>" errors="17" failures="126" tests="2163" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="128" skipped="11" tests="2184" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties> <properties>
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/> <property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
@ -1590,6 +1590,12 @@ at Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (1)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlEncode/string with control char (1)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (x7F)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlEncode/string with control char (x7F)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlWriter writes boolean attributes as true/false" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="XmlWriter writes boolean attributes as true/false" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="a succeeding test can still be skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="analyse no analysis" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="analyse no analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="array&lt;int, N> -> toString" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="array&lt;int, N> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="benchmark function call/without chronometer" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="benchmark function call/without chronometer" time="{duration}" status="run"/>
@ -1624,12 +1630,67 @@ at Misc.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="comparisons between const int variables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="comparisons between const int variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="comparisons between int variables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="comparisons between int variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="convertToBits" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="convertToBits" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="dynamic skipping works with generators" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 41
at Skip.tests.cpp:<line number>
</skipped>
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="empty tags are not allowed" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="empty tags are not allowed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="erfc_inv" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="erfc_inv" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="estimate_clock_resolution" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="estimate_clock_resolution" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/d (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/d (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/e (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/e (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/f (leaf)" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="even more nested SECTION tests/f (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="failed assertions before SKIP cause test case to fail" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure message="3 == 4" type="CHECK">
FAILED:
CHECK( 3 == 4 )
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing for some generator values causes entire test case to fail" time="{duration}" status="run">
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/not skipped" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="is_unary_function" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="is_unary_function" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="just failure" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="just failure" time="{duration}" status="run">
<failure type="FAIL"> <failure type="FAIL">
@ -1742,6 +1803,19 @@ at Misc.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/less than" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/less than" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B2/B" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B" time="{duration}" status="run">
<system-out>
a!
b1!
!
</system-out>
</testcase>
<testcase classname="<exe-name>.global" name="non streamable - with conv. op" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="non streamable - with conv. op" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="non-copyable objects" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="non-copyable objects" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="normal_cdf" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="normal_cdf" time="{duration}" status="run"/>
@ -1793,6 +1867,14 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="resolution" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="resolution" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, chronometer" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="run_for_at_least, chronometer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, int" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="run_for_at_least, int" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/not skipped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/also not skipped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="send a single char to INFO" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="send a single char to INFO" time="{duration}" status="run">
<failure message="false" type="REQUIRE"> <failure message="false" type="REQUIRE">
FAILED: FAILED:
@ -1811,6 +1893,13 @@ at Message.tests.cpp:<line number>
</failure> </failure>
</testcase> </testcase>
<testcase classname="<exe-name>.global" name="shortened hide tags are split apart" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="shortened hide tags are split apart" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="skipped tests can optionally provide a reason" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="splitString" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="splitString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stacks unscoped info in loops" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="stacks unscoped info in loops" time="{duration}" status="run">
<failure message="false" type="CHECK"> <failure message="false" type="CHECK">
@ -1855,6 +1944,12 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="strlen3" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="strlen3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tables" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="tables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tags with dots in later positions are not parsed as hidden" time="{duration}" status="run"/> <testcase classname="<exe-name>.global" name="tags with dots in later positions are not parsed as hidden" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tests can be skipped dynamically at runtime" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="thrown std::strings are translated" time="{duration}" status="run"> <testcase classname="<exe-name>.global" name="thrown std::strings are translated" time="{duration}" status="run">
<error type="TEST_CASE"> <error type="TEST_CASE">
FAILED: FAILED:
@ -1904,6 +1999,9 @@ This would not be caught previously
A string sent directly to stdout A string sent directly to stdout
Message from section one Message from section one
Message from section two Message from section two
a!
b1!
!
</system-out> </system-out>
<system-err> <system-err>
Nor would this Nor would this

89
tests/SelfTest/Baselines/sonarqube.sw.approved.txt
View File

@ -1832,6 +1832,95 @@ at Misc.tests.cpp:<line number>
<testCase name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" duration="{duration}"/> <testCase name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" duration="{duration}"/>
<testCase name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" duration="{duration}"/> <testCase name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" duration="{duration}"/>
</file> </file>
<file path="tests/<exe-name>/UsageTests/Skip.tests.cpp">
<testCase name="a succeeding test can still be skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="dynamic skipping works with generators" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
skipping because answer = 41
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failed assertions before SKIP cause test case to fail" duration="{duration}">
<skipped message="CHECK(3 == 4)">
FAILED:
CHECK( 3 == 4 )
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing for some generator values causes entire test case to fail" duration="{duration}">
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing in some unskipped sections causes entire test case to fail/skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing in some unskipped sections causes entire test case to fail/not skipped" duration="{duration}">
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="nested sections can be skipped dynamically at runtime/B2/B" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="nested sections can be skipped dynamically at runtime/B" duration="{duration}"/>
<testCase name="sections can be skipped dynamically at runtime/not skipped" duration="{duration}"/>
<testCase name="sections can be skipped dynamically at runtime/skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="sections can be skipped dynamically at runtime/also not skipped" duration="{duration}"/>
<testCase name="skipped tests can optionally provide a reason" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="tests can be skipped dynamically at runtime" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
</file>
<file path="tests/<exe-name>/UsageTests/ToStringChrono.tests.cpp"> <file path="tests/<exe-name>/UsageTests/ToStringChrono.tests.cpp">
<testCase name="Stringifying std::chrono::duration helpers" duration="{duration}"/> <testCase name="Stringifying std::chrono::duration helpers" duration="{duration}"/>
<testCase name="Stringifying std::chrono::duration with weird ratios" duration="{duration}"/> <testCase name="Stringifying std::chrono::duration with weird ratios" duration="{duration}"/>

89
tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt
View File

@ -1831,6 +1831,95 @@ at Misc.tests.cpp:<line number>
<testCase name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" duration="{duration}"/> <testCase name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" duration="{duration}"/>
<testCase name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" duration="{duration}"/> <testCase name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" duration="{duration}"/>
</file> </file>
<file path="tests/<exe-name>/UsageTests/Skip.tests.cpp">
<testCase name="a succeeding test can still be skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="dynamic skipping works with generators" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
skipping because answer = 41
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failed assertions before SKIP cause test case to fail" duration="{duration}">
<skipped message="CHECK(3 == 4)">
FAILED:
CHECK( 3 == 4 )
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing for some generator values causes entire test case to fail" duration="{duration}">
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing in some unskipped sections causes entire test case to fail/skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="failing in some unskipped sections causes entire test case to fail/not skipped" duration="{duration}">
<skipped message="FAIL()">
FAILED:
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="nested sections can be skipped dynamically at runtime/B2/B" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="nested sections can be skipped dynamically at runtime/B" duration="{duration}"/>
<testCase name="sections can be skipped dynamically at runtime/not skipped" duration="{duration}"/>
<testCase name="sections can be skipped dynamically at runtime/skipped" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="sections can be skipped dynamically at runtime/also not skipped" duration="{duration}"/>
<testCase name="skipped tests can optionally provide a reason" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
<testCase name="tests can be skipped dynamically at runtime" duration="{duration}">
<skipped message="SKIP()">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testCase>
</file>
<file path="tests/<exe-name>/UsageTests/ToStringChrono.tests.cpp"> <file path="tests/<exe-name>/UsageTests/ToStringChrono.tests.cpp">
<testCase name="Stringifying std::chrono::duration helpers" duration="{duration}"/> <testCase name="Stringifying std::chrono::duration helpers" duration="{duration}"/>
<testCase name="Stringifying std::chrono::duration with weird ratios" duration="{duration}"/> <testCase name="Stringifying std::chrono::duration with weird ratios" duration="{duration}"/>

43
tests/SelfTest/Baselines/tap.sw.approved.txt
View File

@ -3609,6 +3609,10 @@ ok {test-number} - encode( "[\x01]" ) == "[\\x01]" for: "[\x01]" == "[\x01]"
ok {test-number} - encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F]" == "[\x7F]" ok {test-number} - encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F]" == "[\x7F]"
# XmlWriter writes boolean attributes as true/false # XmlWriter writes boolean attributes as true/false
ok {test-number} - stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> <Element1 attr1="true" attr2="false"/> " ( contains: "attr1="true"" and contains: "attr2="false"" ) ok {test-number} - stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> <Element1 attr1="true" attr2="false"/> " ( contains: "attr1="true"" and contains: "attr2="false"" )
# a succeeding test can still be skipped
ok {test-number} -
# a succeeding test can still be skipped
ok {test-number} - # SKIP
# analyse no analysis # analyse no analysis
ok {test-number} - analysis.mean.point.count() == 23 for: 23.0 == 23 ok {test-number} - analysis.mean.point.count() == 23 for: 23.0 == 23
# analyse no analysis # analyse no analysis
@ -3779,6 +3783,12 @@ ok {test-number} - convertToBits( -0. ) == ( 1ULL << 63 ) for: 92233720368547758
ok {test-number} - convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1 ok {test-number} - convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1
# convertToBits # convertToBits
ok {test-number} - convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1 ok {test-number} - convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1
# dynamic skipping works with generators
ok {test-number} - # SKIP 'skipping because answer = 41'
# dynamic skipping works with generators
ok {test-number} -
# dynamic skipping works with generators
ok {test-number} - # SKIP 'skipping because answer = 43'
# empty tags are not allowed # empty tags are not allowed
ok {test-number} - Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo) ok {test-number} - Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo)
# erfc_inv # erfc_inv
@ -3797,6 +3807,22 @@ ok {test-number} -
ok {test-number} - ok {test-number} -
# even more nested SECTION tests # even more nested SECTION tests
ok {test-number} - ok {test-number} -
# failed assertions before SKIP cause test case to fail
not ok {test-number} - 3 == 4
# failed assertions before SKIP cause test case to fail
ok {test-number} - # SKIP
# failing for some generator values causes entire test case to fail
not ok {test-number} - explicitly
# failing for some generator values causes entire test case to fail
ok {test-number} - # SKIP
# failing for some generator values causes entire test case to fail
not ok {test-number} - explicitly
# failing for some generator values causes entire test case to fail
ok {test-number} - # SKIP
# failing in some unskipped sections causes entire test case to fail
ok {test-number} - # SKIP
# failing in some unskipped sections causes entire test case to fail
not ok {test-number} - explicitly
loose text artifact loose text artifact
# is_unary_function # is_unary_function
ok {test-number} - with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value' ok {test-number} - with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value'
@ -3906,6 +3932,11 @@ ok {test-number} - a != b for: 1 != 2
ok {test-number} - b != a for: 2 != 1 ok {test-number} - b != a for: 2 != 1
# nested SECTION tests # nested SECTION tests
ok {test-number} - a != b for: 1 != 2 ok {test-number} - a != b for: 1 != 2
a!
b1!
# nested sections can be skipped dynamically at runtime
ok {test-number} - # SKIP
!
# non streamable - with conv. op # non streamable - with conv. op
ok {test-number} - s == "7" for: "7" == "7" ok {test-number} - s == "7" for: "7" == "7"
# non-copyable objects # non-copyable objects
@ -4070,12 +4101,20 @@ ok {test-number} - Timing.elapsed >= time for: 128 ns >= 100 ns
ok {test-number} - Timing.result == Timing.iterations + 17 for: 145 == 145 ok {test-number} - Timing.result == Timing.iterations + 17 for: 145 == 145
# run_for_at_least, int # run_for_at_least, int
ok {test-number} - Timing.iterations >= time.count() for: 128 >= 100 ok {test-number} - Timing.iterations >= time.count() for: 128 >= 100
# sections can be skipped dynamically at runtime
ok {test-number} -
# sections can be skipped dynamically at runtime
ok {test-number} - # SKIP
# sections can be skipped dynamically at runtime
ok {test-number} -
# send a single char to INFO # send a single char to INFO
not ok {test-number} - false with 1 message: '3' not ok {test-number} - false with 1 message: '3'
# sends information to INFO # sends information to INFO
not ok {test-number} - false with 2 messages: 'hi' and 'i := 7' not ok {test-number} - false with 2 messages: 'hi' and 'i := 7'
# shortened hide tags are split apart # shortened hide tags are split apart
ok {test-number} - testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} ) ok {test-number} - testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
# skipped tests can optionally provide a reason
ok {test-number} - # SKIP 'skipping because answer = 43'
# splitString # splitString
ok {test-number} - splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { } ok {test-number} - splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
# splitString # splitString
@ -4158,6 +4197,8 @@ ok {test-number} - strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(
ok {test-number} - testcase.tags.size() == 1 for: 1 == 1 ok {test-number} - testcase.tags.size() == 1 for: 1 == 1
# tags with dots in later positions are not parsed as hidden # tags with dots in later positions are not parsed as hidden
ok {test-number} - testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag ok {test-number} - testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag
# tests can be skipped dynamically at runtime
ok {test-number} - # SKIP
# thrown std::strings are translated # thrown std::strings are translated
not ok {test-number} - unexpected exception with message: 'Why would you throw a std::string?' not ok {test-number} - unexpected exception with message: 'Why would you throw a std::string?'
# toString on const wchar_t const pointer returns the string contents # toString on const wchar_t const pointer returns the string contents
@ -4330,5 +4371,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} - ok {test-number} -
# xmlentitycheck # xmlentitycheck
ok {test-number} - ok {test-number} -
1..2163 1..2184

40
tests/SelfTest/Baselines/tap.sw.multi.approved.txt
View File

@ -3602,6 +3602,10 @@ ok {test-number} - encode( "[\x01]" ) == "[\\x01]" for: "[\x01]" == "[\x01]"
ok {test-number} - encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F]" == "[\x7F]" ok {test-number} - encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F]" == "[\x7F]"
# XmlWriter writes boolean attributes as true/false # XmlWriter writes boolean attributes as true/false
ok {test-number} - stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> <Element1 attr1="true" attr2="false"/> " ( contains: "attr1="true"" and contains: "attr2="false"" ) ok {test-number} - stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?> <Element1 attr1="true" attr2="false"/> " ( contains: "attr1="true"" and contains: "attr2="false"" )
# a succeeding test can still be skipped
ok {test-number} -
# a succeeding test can still be skipped
ok {test-number} - # SKIP
# analyse no analysis # analyse no analysis
ok {test-number} - analysis.mean.point.count() == 23 for: 23.0 == 23 ok {test-number} - analysis.mean.point.count() == 23 for: 23.0 == 23
# analyse no analysis # analyse no analysis
@ -3772,6 +3776,12 @@ ok {test-number} - convertToBits( -0. ) == ( 1ULL << 63 ) for: 92233720368547758
ok {test-number} - convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1 ok {test-number} - convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1
# convertToBits # convertToBits
ok {test-number} - convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1 ok {test-number} - convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1
# dynamic skipping works with generators
ok {test-number} - # SKIP 'skipping because answer = 41'
# dynamic skipping works with generators
ok {test-number} -
# dynamic skipping works with generators
ok {test-number} - # SKIP 'skipping because answer = 43'
# empty tags are not allowed # empty tags are not allowed
ok {test-number} - Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo) ok {test-number} - Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo)
# erfc_inv # erfc_inv
@ -3790,6 +3800,22 @@ ok {test-number} -
ok {test-number} - ok {test-number} -
# even more nested SECTION tests # even more nested SECTION tests
ok {test-number} - ok {test-number} -
# failed assertions before SKIP cause test case to fail
not ok {test-number} - 3 == 4
# failed assertions before SKIP cause test case to fail
ok {test-number} - # SKIP
# failing for some generator values causes entire test case to fail
not ok {test-number} - explicitly
# failing for some generator values causes entire test case to fail
ok {test-number} - # SKIP
# failing for some generator values causes entire test case to fail
not ok {test-number} - explicitly
# failing for some generator values causes entire test case to fail
ok {test-number} - # SKIP
# failing in some unskipped sections causes entire test case to fail
ok {test-number} - # SKIP
# failing in some unskipped sections causes entire test case to fail
not ok {test-number} - explicitly
# is_unary_function # is_unary_function
ok {test-number} - with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value' ok {test-number} - with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value'
# is_unary_function # is_unary_function
@ -3898,6 +3924,8 @@ ok {test-number} - a != b for: 1 != 2
ok {test-number} - b != a for: 2 != 1 ok {test-number} - b != a for: 2 != 1
# nested SECTION tests # nested SECTION tests
ok {test-number} - a != b for: 1 != 2 ok {test-number} - a != b for: 1 != 2
# nested sections can be skipped dynamically at runtime
ok {test-number} - # SKIP
# non streamable - with conv. op # non streamable - with conv. op
ok {test-number} - s == "7" for: "7" == "7" ok {test-number} - s == "7" for: "7" == "7"
# non-copyable objects # non-copyable objects
@ -4062,12 +4090,20 @@ ok {test-number} - Timing.elapsed >= time for: 128 ns >= 100 ns
ok {test-number} - Timing.result == Timing.iterations + 17 for: 145 == 145 ok {test-number} - Timing.result == Timing.iterations + 17 for: 145 == 145
# run_for_at_least, int # run_for_at_least, int
ok {test-number} - Timing.iterations >= time.count() for: 128 >= 100 ok {test-number} - Timing.iterations >= time.count() for: 128 >= 100
# sections can be skipped dynamically at runtime
ok {test-number} -
# sections can be skipped dynamically at runtime
ok {test-number} - # SKIP
# sections can be skipped dynamically at runtime
ok {test-number} -
# send a single char to INFO # send a single char to INFO
not ok {test-number} - false with 1 message: '3' not ok {test-number} - false with 1 message: '3'
# sends information to INFO # sends information to INFO
not ok {test-number} - false with 2 messages: 'hi' and 'i := 7' not ok {test-number} - false with 2 messages: 'hi' and 'i := 7'
# shortened hide tags are split apart # shortened hide tags are split apart
ok {test-number} - testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} ) ok {test-number} - testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
# skipped tests can optionally provide a reason
ok {test-number} - # SKIP 'skipping because answer = 43'
# splitString # splitString
ok {test-number} - splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { } ok {test-number} - splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
# splitString # splitString
@ -4150,6 +4186,8 @@ ok {test-number} - strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(
ok {test-number} - testcase.tags.size() == 1 for: 1 == 1 ok {test-number} - testcase.tags.size() == 1 for: 1 == 1
# tags with dots in later positions are not parsed as hidden # tags with dots in later positions are not parsed as hidden
ok {test-number} - testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag ok {test-number} - testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag
# tests can be skipped dynamically at runtime
ok {test-number} - # SKIP
# thrown std::strings are translated # thrown std::strings are translated
not ok {test-number} - unexpected exception with message: 'Why would you throw a std::string?' not ok {test-number} - unexpected exception with message: 'Why would you throw a std::string?'
# toString on const wchar_t const pointer returns the string contents # toString on const wchar_t const pointer returns the string contents
@ -4322,5 +4360,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} - ok {test-number} -
# xmlentitycheck # xmlentitycheck
ok {test-number} - ok {test-number} -
1..2163 1..2184

34
tests/SelfTest/Baselines/teamcity.sw.approved.txt
View File

@ -722,6 +722,9 @@
##teamcity[testFinished name='XmlEncode' duration="{duration}"] ##teamcity[testFinished name='XmlEncode' duration="{duration}"]
##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false'] ##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false']
##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"] ##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"]
##teamcity[testStarted name='a succeeding test can still be skipped']
##teamcity[testIgnored name='a succeeding test can still be skipped' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='a succeeding test can still be skipped' duration="{duration}"]
##teamcity[testStarted name='analyse no analysis'] ##teamcity[testStarted name='analyse no analysis']
##teamcity[testFinished name='analyse no analysis' duration="{duration}"] ##teamcity[testFinished name='analyse no analysis' duration="{duration}"]
##teamcity[testStarted name='array<int, N> -> toString'] ##teamcity[testStarted name='array<int, N> -> toString']
@ -748,6 +751,10 @@
##teamcity[testFinished name='comparisons between int variables' duration="{duration}"] ##teamcity[testFinished name='comparisons between int variables' duration="{duration}"]
##teamcity[testStarted name='convertToBits'] ##teamcity[testStarted name='convertToBits']
##teamcity[testFinished name='convertToBits' duration="{duration}"] ##teamcity[testFinished name='convertToBits' duration="{duration}"]
##teamcity[testStarted name='dynamic skipping works with generators']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 41"']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='dynamic skipping works with generators' duration="{duration}"]
##teamcity[testStarted name='empty tags are not allowed'] ##teamcity[testStarted name='empty tags are not allowed']
##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"] ##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"]
##teamcity[testStarted name='erfc_inv'] ##teamcity[testStarted name='erfc_inv']
@ -756,6 +763,20 @@
##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"] ##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"]
##teamcity[testStarted name='even more nested SECTION tests'] ##teamcity[testStarted name='even more nested SECTION tests']
##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='failed assertions before SKIP cause test case to fail']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexpression failed|n CHECK( 3 == 4 )|nwith expansion:|n 3 == 4|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failed assertions before SKIP cause test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing for some generator values causes entire test case to fail']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failing for some generator values causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing in some unskipped sections causes entire test case to fail']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nnot skipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='failing in some unskipped sections causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='first tag'] ##teamcity[testStarted name='first tag']
##teamcity[testFinished name='first tag' duration="{duration}"] ##teamcity[testFinished name='first tag' duration="{duration}"]
##teamcity[testStarted name='has printf'] ##teamcity[testStarted name='has printf']
@ -802,6 +823,10 @@ loose text artifact
##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested SECTION tests'] ##teamcity[testStarted name='nested SECTION tests']
##teamcity[testFinished name='nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='nested sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nB|nB2|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testStdOut name='nested sections can be skipped dynamically at runtime' out='a!|nb1!|n!|n']
##teamcity[testFinished name='nested sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='non streamable - with conv. op'] ##teamcity[testStarted name='non streamable - with conv. op']
##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"] ##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"]
##teamcity[testStarted name='non-copyable objects'] ##teamcity[testStarted name='non-copyable objects']
@ -847,6 +872,9 @@ loose text artifact
##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"] ##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"]
##teamcity[testStarted name='second tag'] ##teamcity[testStarted name='second tag']
##teamcity[testFinished name='second tag' duration="{duration}"] ##teamcity[testFinished name='second tag' duration="{duration}"]
##teamcity[testStarted name='sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='send a single char to INFO'] ##teamcity[testStarted name='send a single char to INFO']
##teamcity[testFailed name='send a single char to INFO' message='Misc.tests.cpp:<line number>|n...............................................................................|n|nMisc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n'] ##teamcity[testFailed name='send a single char to INFO' message='Misc.tests.cpp:<line number>|n...............................................................................|n|nMisc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='send a single char to INFO' duration="{duration}"] ##teamcity[testFinished name='send a single char to INFO' duration="{duration}"]
@ -855,6 +883,9 @@ loose text artifact
##teamcity[testFinished name='sends information to INFO' duration="{duration}"] ##teamcity[testFinished name='sends information to INFO' duration="{duration}"]
##teamcity[testStarted name='shortened hide tags are split apart'] ##teamcity[testStarted name='shortened hide tags are split apart']
##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"] ##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"]
##teamcity[testStarted name='skipped tests can optionally provide a reason']
##teamcity[testIgnored name='skipped tests can optionally provide a reason' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='skipped tests can optionally provide a reason' duration="{duration}"]
##teamcity[testStarted name='splitString'] ##teamcity[testStarted name='splitString']
##teamcity[testFinished name='splitString' duration="{duration}"] ##teamcity[testFinished name='splitString' duration="{duration}"]
##teamcity[testStarted name='stacks unscoped info in loops'] ##teamcity[testStarted name='stacks unscoped info in loops']
@ -899,6 +930,9 @@ loose text artifact
##teamcity[testFinished name='tables' duration="{duration}"] ##teamcity[testFinished name='tables' duration="{duration}"]
##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden'] ##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden']
##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"] ##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"]
##teamcity[testStarted name='tests can be skipped dynamically at runtime']
##teamcity[testIgnored name='tests can be skipped dynamically at runtime' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='tests can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='thrown std::strings are translated'] ##teamcity[testStarted name='thrown std::strings are translated']
##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:<line number>|n...............................................................................|n|nException.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"'] ##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:<line number>|n...............................................................................|n|nException.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"']
##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"] ##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"]

34
tests/SelfTest/Baselines/teamcity.sw.multi.approved.txt
View File

@ -722,6 +722,9 @@
##teamcity[testFinished name='XmlEncode' duration="{duration}"] ##teamcity[testFinished name='XmlEncode' duration="{duration}"]
##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false'] ##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false']
##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"] ##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"]
##teamcity[testStarted name='a succeeding test can still be skipped']
##teamcity[testIgnored name='a succeeding test can still be skipped' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='a succeeding test can still be skipped' duration="{duration}"]
##teamcity[testStarted name='analyse no analysis'] ##teamcity[testStarted name='analyse no analysis']
##teamcity[testFinished name='analyse no analysis' duration="{duration}"] ##teamcity[testFinished name='analyse no analysis' duration="{duration}"]
##teamcity[testStarted name='array<int, N> -> toString'] ##teamcity[testStarted name='array<int, N> -> toString']
@ -748,6 +751,10 @@
##teamcity[testFinished name='comparisons between int variables' duration="{duration}"] ##teamcity[testFinished name='comparisons between int variables' duration="{duration}"]
##teamcity[testStarted name='convertToBits'] ##teamcity[testStarted name='convertToBits']
##teamcity[testFinished name='convertToBits' duration="{duration}"] ##teamcity[testFinished name='convertToBits' duration="{duration}"]
##teamcity[testStarted name='dynamic skipping works with generators']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 41"']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='dynamic skipping works with generators' duration="{duration}"]
##teamcity[testStarted name='empty tags are not allowed'] ##teamcity[testStarted name='empty tags are not allowed']
##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"] ##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"]
##teamcity[testStarted name='erfc_inv'] ##teamcity[testStarted name='erfc_inv']
@ -756,6 +763,20 @@
##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"] ##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"]
##teamcity[testStarted name='even more nested SECTION tests'] ##teamcity[testStarted name='even more nested SECTION tests']
##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='failed assertions before SKIP cause test case to fail']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexpression failed|n CHECK( 3 == 4 )|nwith expansion:|n 3 == 4|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failed assertions before SKIP cause test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing for some generator values causes entire test case to fail']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failing for some generator values causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing in some unskipped sections causes entire test case to fail']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nnot skipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='failing in some unskipped sections causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='first tag'] ##teamcity[testStarted name='first tag']
##teamcity[testFinished name='first tag' duration="{duration}"] ##teamcity[testFinished name='first tag' duration="{duration}"]
##teamcity[testStarted name='has printf'] ##teamcity[testStarted name='has printf']
@ -801,6 +822,10 @@
##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested SECTION tests'] ##teamcity[testStarted name='nested SECTION tests']
##teamcity[testFinished name='nested SECTION tests' duration="{duration}"] ##teamcity[testFinished name='nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='nested sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nB|nB2|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testStdOut name='nested sections can be skipped dynamically at runtime' out='a!|nb1!|n!|n']
##teamcity[testFinished name='nested sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='non streamable - with conv. op'] ##teamcity[testStarted name='non streamable - with conv. op']
##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"] ##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"]
##teamcity[testStarted name='non-copyable objects'] ##teamcity[testStarted name='non-copyable objects']
@ -846,6 +871,9 @@
##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"] ##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"]
##teamcity[testStarted name='second tag'] ##teamcity[testStarted name='second tag']
##teamcity[testFinished name='second tag' duration="{duration}"] ##teamcity[testFinished name='second tag' duration="{duration}"]
##teamcity[testStarted name='sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='send a single char to INFO'] ##teamcity[testStarted name='send a single char to INFO']
##teamcity[testFailed name='send a single char to INFO' message='Misc.tests.cpp:<line number>|n...............................................................................|n|nMisc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n'] ##teamcity[testFailed name='send a single char to INFO' message='Misc.tests.cpp:<line number>|n...............................................................................|n|nMisc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='send a single char to INFO' duration="{duration}"] ##teamcity[testFinished name='send a single char to INFO' duration="{duration}"]
@ -854,6 +882,9 @@
##teamcity[testFinished name='sends information to INFO' duration="{duration}"] ##teamcity[testFinished name='sends information to INFO' duration="{duration}"]
##teamcity[testStarted name='shortened hide tags are split apart'] ##teamcity[testStarted name='shortened hide tags are split apart']
##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"] ##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"]
##teamcity[testStarted name='skipped tests can optionally provide a reason']
##teamcity[testIgnored name='skipped tests can optionally provide a reason' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='skipped tests can optionally provide a reason' duration="{duration}"]
##teamcity[testStarted name='splitString'] ##teamcity[testStarted name='splitString']
##teamcity[testFinished name='splitString' duration="{duration}"] ##teamcity[testFinished name='splitString' duration="{duration}"]
##teamcity[testStarted name='stacks unscoped info in loops'] ##teamcity[testStarted name='stacks unscoped info in loops']
@ -898,6 +929,9 @@
##teamcity[testFinished name='tables' duration="{duration}"] ##teamcity[testFinished name='tables' duration="{duration}"]
##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden'] ##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden']
##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"] ##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"]
##teamcity[testStarted name='tests can be skipped dynamically at runtime']
##teamcity[testIgnored name='tests can be skipped dynamically at runtime' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='tests can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='thrown std::strings are translated'] ##teamcity[testStarted name='thrown std::strings are translated']
##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:<line number>|n...............................................................................|n|nException.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"'] ##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:<line number>|n...............................................................................|n|nException.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"']
##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"] ##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"]

2198
tests/SelfTest/Baselines/xml.sw.approved.txt
View File

File diff suppressed because it is too large Load Diff

2198
tests/SelfTest/Baselines/xml.sw.multi.approved.txt
View File

File diff suppressed because it is too large Load Diff

73
tests/SelfTest/UsageTests/Skip.tests.cpp Normal file
View File

@ -0,0 +1,73 @@
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators_range.hpp>
#include <iostream>
TEST_CASE( "tests can be skipped dynamically at runtime", "[skipping]" ) {
SKIP();
FAIL( "this is not reached" );
}
TEST_CASE( "skipped tests can optionally provide a reason", "[skipping]" ) {
const int answer = 43;
SKIP( "skipping because answer = " << answer );
FAIL( "this is not reached" );
}
TEST_CASE( "sections can be skipped dynamically at runtime", "[skipping]" ) {
SECTION( "not skipped" ) { SUCCEED(); }
SECTION( "skipped" ) { SKIP(); }
SECTION( "also not skipped" ) { SUCCEED(); }
}
TEST_CASE( "nested sections can be skipped dynamically at runtime",
"[skipping]" ) {
SECTION( "A" ) { std::cout << "a"; }
SECTION( "B" ) {
SECTION( "B1" ) { std::cout << "b1"; }
SECTION( "B2" ) { SKIP(); }
}
std::cout << "!\n";
}
TEST_CASE( "dynamic skipping works with generators", "[skipping]" ) {
const int answer = GENERATE( 41, 42, 43 );
if ( answer != 42 ) { SKIP( "skipping because answer = " << answer ); }
SUCCEED();
}
TEST_CASE( "failed assertions before SKIP cause test case to fail",
"[skipping][!shouldfail]" ) {
CHECK( 3 == 4 );
SKIP();
}
TEST_CASE( "a succeeding test can still be skipped",
"[skipping][!shouldfail]" ) {
SUCCEED();
SKIP();
}
TEST_CASE( "failing in some unskipped sections causes entire test case to fail",
"[skipping][!shouldfail]" ) {
SECTION( "skipped" ) { SKIP(); }
SECTION( "not skipped" ) { FAIL(); }
}
TEST_CASE( "failing for some generator values causes entire test case to fail",
"[skipping][!shouldfail]" ) {
int i = GENERATE( 1, 2, 3, 4 );
if ( i % 2 == 0 ) {
SKIP();
} else {
FAIL();
}
}