3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2025-08-02 13:25:41 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add new SKIP macro for skipping tests at runtime (#2360)

Browse Source 
* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This commit is contained in:
Philip Salzmann
2023-01-12 15:01:47 +01:00
committed by GitHub
parent 52066dbc2a
commit d548be26e3
47 changed files with 3722 additions and 2171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
tests/SelfTest/Baselines/teamcity.sw.approved.txt
View File

@@ -722,6 +722,9 @@
##teamcity[testFinished name='XmlEncode' duration="{duration}"]
##teamcity[testStarted name='XmlWriter writes boolean attributes as true/false']
##teamcity[testFinished name='XmlWriter writes boolean attributes as true/false' duration="{duration}"]
##teamcity[testStarted name='a succeeding test can still be skipped']
##teamcity[testIgnored name='a succeeding test can still be skipped' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='a succeeding test can still be skipped' duration="{duration}"]
##teamcity[testStarted name='analyse no analysis']
##teamcity[testFinished name='analyse no analysis' duration="{duration}"]
##teamcity[testStarted name='array<int, N> -> toString']
@@ -748,6 +751,10 @@
##teamcity[testFinished name='comparisons between int variables' duration="{duration}"]
##teamcity[testStarted name='convertToBits']
##teamcity[testFinished name='convertToBits' duration="{duration}"]
##teamcity[testStarted name='dynamic skipping works with generators']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 41"']
##teamcity[testIgnored name='dynamic skipping works with generators' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='dynamic skipping works with generators' duration="{duration}"]
##teamcity[testStarted name='empty tags are not allowed']
##teamcity[testFinished name='empty tags are not allowed' duration="{duration}"]
##teamcity[testStarted name='erfc_inv']
@@ -756,6 +763,20 @@
##teamcity[testFinished name='estimate_clock_resolution' duration="{duration}"]
##teamcity[testStarted name='even more nested SECTION tests']
##teamcity[testFinished name='even more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='failed assertions before SKIP cause test case to fail']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexpression failed|n CHECK( 3 == 4 )|nwith expansion:|n 3 == 4|n- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failed assertions before SKIP cause test case to fail' message='Skip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failed assertions before SKIP cause test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing for some generator values causes entire test case to fail']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testIgnored name='failing for some generator values causes entire test case to fail' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='failing for some generator values causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='failing in some unskipped sections causes entire test case to fail']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testIgnored name='failing in some unskipped sections causes entire test case to fail' message='-------------------------------------------------------------------------------|nnot skipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit failure- failure ignore as test marked as |'ok to fail|'|n']
##teamcity[testFinished name='failing in some unskipped sections causes entire test case to fail' duration="{duration}"]
##teamcity[testStarted name='first tag']
##teamcity[testFinished name='first tag' duration="{duration}"]
##teamcity[testStarted name='has printf']
@@ -802,6 +823,10 @@ loose text artifact
##teamcity[testFinished name='more nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested SECTION tests']
##teamcity[testFinished name='nested SECTION tests' duration="{duration}"]
##teamcity[testStarted name='nested sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='nested sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nB|nB2|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testStdOut name='nested sections can be skipped dynamically at runtime' out='a!|nb1!|n!|n']
##teamcity[testFinished name='nested sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='non streamable - with conv. op']
##teamcity[testFinished name='non streamable - with conv. op' duration="{duration}"]
##teamcity[testStarted name='non-copyable objects']
@@ -847,6 +872,9 @@ loose text artifact
##teamcity[testFinished name='run_for_at_least, int' duration="{duration}"]
##teamcity[testStarted name='second tag']
##teamcity[testFinished name='second tag' duration="{duration}"]
##teamcity[testStarted name='sections can be skipped dynamically at runtime']
##teamcity[testIgnored name='sections can be skipped dynamically at runtime' message='-------------------------------------------------------------------------------|nskipped|n-------------------------------------------------------------------------------|nSkip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='sections can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='send a single char to INFO']
##teamcity[testFailed name='send a single char to INFO' message='Misc.tests.cpp:<line number>|n...............................................................................|n|nMisc.tests.cpp:<line number>|nexpression failed with message:|n "3"|n REQUIRE( false )|nwith expansion:|n false|n']
##teamcity[testFinished name='send a single char to INFO' duration="{duration}"]
@@ -855,6 +883,9 @@ loose text artifact
##teamcity[testFinished name='sends information to INFO' duration="{duration}"]
##teamcity[testStarted name='shortened hide tags are split apart']
##teamcity[testFinished name='shortened hide tags are split apart' duration="{duration}"]
##teamcity[testStarted name='skipped tests can optionally provide a reason']
##teamcity[testIgnored name='skipped tests can optionally provide a reason' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip with message:|n "skipping because answer = 43"']
##teamcity[testFinished name='skipped tests can optionally provide a reason' duration="{duration}"]
##teamcity[testStarted name='splitString']
##teamcity[testFinished name='splitString' duration="{duration}"]
##teamcity[testStarted name='stacks unscoped info in loops']
@@ -899,6 +930,9 @@ loose text artifact
##teamcity[testFinished name='tables' duration="{duration}"]
##teamcity[testStarted name='tags with dots in later positions are not parsed as hidden']
##teamcity[testFinished name='tags with dots in later positions are not parsed as hidden' duration="{duration}"]
##teamcity[testStarted name='tests can be skipped dynamically at runtime']
##teamcity[testIgnored name='tests can be skipped dynamically at runtime' message='Skip.tests.cpp:<line number>|n...............................................................................|n|nSkip.tests.cpp:<line number>|nexplicit skip']
##teamcity[testFinished name='tests can be skipped dynamically at runtime' duration="{duration}"]
##teamcity[testStarted name='thrown std::strings are translated']
##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:<line number>|n...............................................................................|n|nException.tests.cpp:<line number>|nunexpected exception with message:|n "Why would you throw a std::string?"']
##teamcity[testFinished name='thrown std::strings are translated' duration="{duration}"]