3.7 KiB

Raw Permalink Blame History

Explicitly skipping, passing, and failing tests at runtime

Skipping Test Cases at Runtime

Introduced in Catch2 3.3.0.

In some situations it may not be possible to meaningfully execute a test case, for example when the system under test is missing certain hardware capabilities. If the required conditions can only be determined at runtime, it often doesn't make sense to consider such a test case as either passed or failed, because it simply cannot run at all.

To properly express such scenarios, Catch2 provides a way to explicitly skip test cases, using the SKIP macro:

SKIP( [streamable expression] )

Example usage:

TEST_CASE("copy files between drives") {
    if(getNumberOfHardDrives() < 2) {
        SKIP("at least two hard drives required");
    }
    // ...
}

This test case is then reported as skipped instead of passed or failed.

The SKIP macro behaves similarly to an explicit FAIL, in that it is the last expression that will be executed:

TEST_CASE("my test") {
    printf("foo");
    SKIP();
    printf("bar"); // not printed
}

However a failed assertion before a SKIP still causes the entire test case to fail:

TEST_CASE("failing test") {
    CHECK(1 == 2);
    SKIP();
}

Interaction with Sections and Generators

Sections, nested sections as well as specific outputs from generators can all be individually skipped, with the rest executing as usual:

TEST_CASE("complex test case") {
  int value = GENERATE(2, 4, 6);
  SECTION("a") {
    SECTION("a1") { CHECK(value < 8); }
    SECTION("a2") {
      if (value == 4) {
        SKIP();
      }
      CHECK(value % 2 == 0);
    }
  }
}

This test case will report 5 passing assertions; one for each of the three values in section a1, and then two in section a2, from values 2 and 4.

Note that as soon as one section is skipped, the entire test case will be reported as skipped (unless there is a failing assertion, in which case the test is handled as failed instead).

Note that if all test cases in a run are skipped, Catch2 returns a non-zero exit code, same as it does if no test cases have run. This behaviour can be overridden using the --allow-running-no-tests flag.

`SKIP` inside generators

You can also use the SKIP macro inside generator's constructor to handle cases where the generator is empty, but you do not want to fail the test case.

Passing and failing test cases

Test cases can also be explicitly passed or failed, without the use of assertions, and with a specific message. This can be useful to handle complex preconditions/postconditions and give useful error messages when they fail.

SUCCEED( [streamable expression] )

SUCCEED is morally equivalent with INFO( [streamable expression] ); REQUIRE( true );. Note that it does not stop further test execution, so it cannot be used to guard failing assertions from being executed.

In practice, SUCCEED is usually used as a test placeholder, to avoid failing a test case due to missing assertions.

TEST_CASE( "SUCCEED showcase" ) {
    int I = 1;
    SUCCEED( "I is " << I );
    // ... execution continues here ...
}

FAIL( [streamable expression] )

FAIL is morally equivalent with INFO( [streamable expression] ); REQUIRE( false );.

In practice, FAIL is usually used to stop executing test that is currently known to be broken, but has to be fixed later.

TEST_CASE( "FAIL showcase" ) {
    FAIL( "This test case causes segfault, which breaks CI." );
    // ... this will not be executed ...
}

Home

3.7 KiB Raw Permalink Blame History