mirror of
https://github.com/catchorg/Catch2.git
synced 2025-08-03 05:45:39 +02:00
Add new SKIP macro for skipping tests at runtime (#2360)
* Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This commit is contained in:
Philip Salzmann
@@ -14659,6 +14659,16 @@ with expansion:
|
||||
<Element1 attr1="true" attr2="false"/>
|
||||
" ( contains: "attr1="true"" and contains: "attr2="false"" )
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
a succeeding test can still be skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: PASSED:
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
analyse no analysis
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -15204,6 +15214,34 @@ FloatingPoint.tests.cpp:<line number>: PASSED:
|
||||
with expansion:
|
||||
1 == 1
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
dynamic skipping works with generators
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
explicitly with message:
|
||||
skipping because answer = 41
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
dynamic skipping works with generators
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: PASSED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
dynamic skipping works with generators
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
explicitly with message:
|
||||
skipping because answer = 43
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
empty tags are not allowed
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -15279,6 +15317,67 @@ Misc.tests.cpp:<line number>
|
||||
|
||||
Misc.tests.cpp:<line number>: PASSED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failed assertions before SKIP cause test case to fail
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: FAILED:
|
||||
CHECK( 3 == 4 )
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing for some generator values causes entire test case to fail
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: FAILED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing for some generator values causes entire test case to fail
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing for some generator values causes entire test case to fail
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: FAILED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing for some generator values causes entire test case to fail
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing in some unskipped sections causes entire test case to fail
|
||||
skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
failing in some unskipped sections causes entire test case to fail
|
||||
not skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: FAILED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
first tag
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -15775,6 +15874,40 @@ Misc.tests.cpp:<line number>: PASSED:
|
||||
with expansion:
|
||||
1 != 2
|
||||
|
||||
a-------------------------------------------------------------------------------
|
||||
nested sections can be skipped dynamically at runtime
|
||||
A
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
|
||||
No assertions in section 'A'
|
||||
|
||||
!
|
||||
b1-------------------------------------------------------------------------------
|
||||
nested sections can be skipped dynamically at runtime
|
||||
B
|
||||
B1
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
|
||||
No assertions in section 'B1'
|
||||
|
||||
!
|
||||
-------------------------------------------------------------------------------
|
||||
nested sections can be skipped dynamically at runtime
|
||||
B
|
||||
B2
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
!
|
||||
-------------------------------------------------------------------------------
|
||||
non streamable - with conv. op
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -16376,6 +16509,33 @@ Misc.tests.cpp:<line number>
|
||||
|
||||
No assertions in test case 'second tag'
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
sections can be skipped dynamically at runtime
|
||||
not skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: PASSED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
sections can be skipped dynamically at runtime
|
||||
skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
sections can be skipped dynamically at runtime
|
||||
also not skipped
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: PASSED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
send a single char to INFO
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -16410,6 +16570,16 @@ Tag.tests.cpp:<line number>: PASSED:
|
||||
with expansion:
|
||||
{ {?}, {?} } ( Contains: {?} and Contains: {?} )
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
skipped tests can optionally provide a reason
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
explicitly with message:
|
||||
skipping because answer = 43
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
splitString
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -16837,6 +17007,14 @@ Tag.tests.cpp:<line number>: PASSED:
|
||||
with expansion:
|
||||
magic.tag == magic.tag
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
tests can be skipped dynamically at runtime
|
||||
-------------------------------------------------------------------------------
|
||||
Skip.tests.cpp:<line number>
|
||||
...............................................................................
|
||||
|
||||
Skip.tests.cpp:<line number>: SKIPPED:
|
||||
|
||||
-------------------------------------------------------------------------------
|
||||
thrown std::strings are translated
|
||||
-------------------------------------------------------------------------------
|
||||
@@ -17544,6 +17722,6 @@ Misc.tests.cpp:<line number>
|
||||
Misc.tests.cpp:<line number>: PASSED:
|
||||
|
||||
===============================================================================
|
||||
test cases: 395 | 305 passed | 83 failed | 7 failed as expected
|
||||
assertions: 2163 | 1993 passed | 143 failed | 27 failed as expected
|
||||
test cases: 404 | 305 passed | 84 failed | 5 skipped | 10 failed as expected
|
||||
assertions: 2173 | 1997 passed | 145 failed | 31 failed as expected
|
||||
|
||||
|
||||
Reference in New Issue
Block a user