3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2025-08-03 22:05:39 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add new SKIP macro for skipping tests at runtime (#2360)

Browse Source 
* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This commit is contained in:
Philip Salzmann
2023-01-12 15:01:47 +01:00
committed by GitHub
parent 52066dbc2a
commit d548be26e3
47 changed files with 3722 additions and 2171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
tests/SelfTest/Baselines/compact.sw.approved.txt
View File

@@ -2060,6 +2060,8 @@ Xml.tests.cpp:<line number>: passed: encode( "[\x7F]" ) == "[\\x7F]" for: "[\x7F
Xml.tests.cpp:<line number>: passed: stream.str(), ContainsSubstring(R"(attr1="true")") && ContainsSubstring(R"(attr2="false")") for: "<?xml version="1.0" encoding="UTF-8"?>
<Element1 attr1="true" attr2="false"/>
" ( contains: "attr1="true"" and contains: "attr2="false"" )
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.point.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.lower_bound.count() == 23 for: 23.0 == 23
InternalBenchmark.tests.cpp:<line number>: passed: analysis.mean.upper_bound.count() == 23 for: 23.0 == 23
@@ -2149,6 +2151,9 @@ FloatingPoint.tests.cpp:<line number>: passed: convertToBits( -0. ) == ( 1ULL <<
9223372036854775808 (0x<hex digits>)
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<float>::denorm_min() ) == 1 for: 1 == 1
FloatingPoint.tests.cpp:<line number>: passed: convertToBits( std::numeric_limits<double>::denorm_min() ) == 1 for: 1 == 1
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 41'
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
Tag.tests.cpp:<line number>: passed: Catch::TestCaseInfo("", { "test with an empty tag", "[]" }, dummySourceLineInfo)
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.103560) == Approx(-0.09203687623843015) for: -0.0920368762 == Approx( -0.0920368762 )
InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(1.067400) == Approx(-0.05980291115763361) for: -0.0598029112 == Approx( -0.0598029112 )
@@ -2158,6 +2163,14 @@ InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: failed: 3 == 4
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: failed: explicitly
loose text artifact
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary1)>::value'
Clara.tests.cpp:<line number>: passed: with 1 message: 'Catch::Clara::Detail::is_unary_function<decltype(unary2)>::value'
@@ -2215,6 +2228,10 @@ Misc.tests.cpp:<line number>: passed: a < b for: 1 < 2
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
Misc.tests.cpp:<line number>: passed: b != a for: 2 != 1
Misc.tests.cpp:<line number>: passed: a != b for: 1 != 2
a!
b1!
Skip.tests.cpp:<line number>: skipped:
!
Tricky.tests.cpp:<line number>: passed: s == "7" for: "7" == "7"
Tricky.tests.cpp:<line number>: passed: ti == typeid(int) for: {?} == {?}
InternalBenchmark.tests.cpp:<line number>: passed: normal_cdf(0.000000) == Approx(0.50000000000000000) for: 0.5 == Approx( 0.5 )
@@ -2299,9 +2316,13 @@ InternalBenchmark.tests.cpp:<line number>: passed: x >= old_x for: 128 >= 64
InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed >= time for: 128 ns >= 100 ns
InternalBenchmark.tests.cpp:<line number>: passed: Timing.result == Timing.iterations + 17 for: 145 == 145
InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations >= time.count() for: 128 >= 100
Skip.tests.cpp:<line number>: passed:
Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: failed: false with 1 message: '3'
Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7'
Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc", ','), Equals(std::vector<StringRef>{"abc"}) for: { abc } Equals: { abc }
StringManip.tests.cpp:<line number>: passed: splitStringRef("abc,def", ','), Equals(std::vector<StringRef>{"abc", "def"}) for: { abc, def } Equals: { abc, def }
@@ -2367,6 +2388,7 @@ Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_
Generators.tests.cpp:<line number>: passed: strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)) for: 6 == 6
Tag.tests.cpp:<line number>: passed: testcase.tags.size() == 1 for: 1 == 1
Tag.tests.cpp:<line number>: passed: testcase.tags[0].original == "magic.tag"_catch_sr for: magic.tag == magic.tag
Skip.tests.cpp:<line number>: skipped:
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'Why would you throw a std::string?'
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
Misc.tests.cpp:<line number>: passed: result == "\"wide load\"" for: ""wide load"" == ""wide load""
@@ -2463,7 +2485,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 395 | 305 passed | 83 failed | 7 failed as expected
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected
test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected