3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-04 05:09:53 +01:00
Code Issues Packages Projects Releases Wiki Activity
f797ae7a8f
catch2/tests/CMakeLists.txt

264 lines
12 KiB
CMake
Raw Normal View History

Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
include(MiscFunctions)
Fix VS2017 approvals on AppVeyor Because of a change in VS toolset, missing option <UseFullPaths> is no longer interpreted as "don't pass /FC to the compiler", but rather as "pass /FC to the compiler". This is problematic, because /FC not only changes how much of the path is reporter by the compiler (e.g. in `__FILE__` macro), but it also lower cases the path. This lower-casing of the path broke our approval tests for VS2017 about 5 months ago. Using CMake 3.13 (not yet released) would also let us fix it, but for now we use a vcxproj.user file that is merged with the main project and explicitly disables `/FC`.
2018-08-28 10:48:05 +02:00
####
# Temporary workaround for VS toolset changes in 2017
# We need to disable <UseFullPaths> property, but CMake doesn't support it
# until 3.13 (not yet released)
####
if (MSVC)
Move misc/ to tools/misc
2019-12-06 11:40:53 +01:00
configure_file(${CATCH_DIR}/tools/misc/SelfTest.vcxproj.user
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
${CMAKE_BINARY_DIR}/tests
Fix VS2017 approvals on AppVeyor Because of a change in VS toolset, missing option <UseFullPaths> is no longer interpreted as "don't pass /FC to the compiler", but rather as "pass /FC to the compiler". This is problematic, because /FC not only changes how much of the path is reporter by the compiler (e.g. in `__FILE__` macro), but it also lower cases the path. This lower-casing of the path broke our approval tests for VS2017 about 5 months ago. Using CMake 3.13 (not yet released) would also let us fix it, but for now we use a vcxproj.user file that is merged with the main project and explicitly disables `/FC`.
2018-08-28 10:48:05 +02:00
COPYONLY)
endif(MSVC) #Temporary workaround
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
# define the sources of the self test
# Please keep these ordered alphabetically
set(TEST_SOURCES
Remove catch_default_main.hpp There are two reasons for this: 1) It is highly unlikely that someone has use for this header, which has no customization points and only provides simplest possible main, and cannot link the static library which also provides a default main implementation. 2) It being a header was causing extra complications with the convenience headers, and our checking script. This would either require special handling in the checking script, or would break user's of the main convenience header. All in all, it is simpler and better in the long term to remove it, than to fix its problems.
2020-05-07 16:24:05 +02:00
${SELF_TEST_DIR}/TestRegistrations.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/CmdLine.tests.cpp
Add a test for the internals
2019-08-09 10:50:53 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/Details.tests.cpp
First attempt at data generator support The support is to be considered experimental, that is, the interfaces, the first party generators and helper functions can change or be removed at any point in time. Related to #850
2018-06-12 18:50:39 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/GeneratorsImpl.tests.cpp
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/InternalBenchmark.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/PartTracker.tests.cpp
Introduce Catch's own RNG based on the PCG family of RNGs In the future, we will also want to introduce our own `uniform_int_distribution` and `uniform_real_distribution` to get repeatable test runs across different platforms.
2019-10-06 21:47:54 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/RandomNumberGeneration.tests.cpp
Fix test tag parsing to split [.foo] into [.][foo]
2019-03-29 10:48:56 +01:00
${SELF_TEST_DIR}/IntrospectiveTests/Tag.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/String.tests.cpp
Start adding tests for string manip functions
2019-09-06 18:50:57 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/StringManip.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/Xml.tests.cpp
Moved enum tests that depend on internals to IntrospectiveTests. - also factored out makeEnumInfo, so tests don't need to touch registry - and added usage test that involves namespace
2019-04-25 11:13:11 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/ToString.tests.cpp
Implement a simplified variant of std::unique_ptr<T> This simplified variant supports only a subset of the functionality in `std::unique_ptr<T>`. `Catch::Detail::unique_ptr<T>` only supports single element pointer (no array support) with default deleter. By removing the support for custom deleters, we also avoid requiring significant machinery to support EBO, speeding up instantiations of `unique_ptr<T>` significantly. Catch2 also currently does not need to support `unique_ptr<T[]>`, so that is not supported either.
2020-05-24 23:41:02 +02:00
${SELF_TEST_DIR}/IntrospectiveTests/UniquePtr.tests.cpp
Add --min-duration option A test runner already has a --durations option to print durations. However, this isn't entirely satisfactory. When there are many tests, this produces output spam which makes it hard to find the test failure output. Nevertheless, it is helpful to be informed of tests which are unusually slow. Therefore, introduce a new option --min-duration that causes all durations above a certain threshold to be printed. This allows slow tests to be visible without mentioning every test.
2020-04-13 14:34:27 +02:00
${SELF_TEST_DIR}/TimingTests/Sleep.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/Approx.tests.cpp
${SELF_TEST_DIR}/UsageTests/BDD.tests.cpp
${SELF_TEST_DIR}/UsageTests/Benchmark.tests.cpp
${SELF_TEST_DIR}/UsageTests/Class.tests.cpp
${SELF_TEST_DIR}/UsageTests/Compilation.tests.cpp
${SELF_TEST_DIR}/UsageTests/Condition.tests.cpp
${SELF_TEST_DIR}/UsageTests/Decomposition.tests.cpp
${SELF_TEST_DIR}/UsageTests/EnumToString.tests.cpp
${SELF_TEST_DIR}/UsageTests/Exception.tests.cpp
First attempt at data generator support The support is to be considered experimental, that is, the interfaces, the first party generators and helper functions can change or be removed at any point in time. Related to #850
2018-06-12 18:50:39 +02:00
${SELF_TEST_DIR}/UsageTests/Generators.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/Message.tests.cpp
${SELF_TEST_DIR}/UsageTests/Misc.tests.cpp
Add `std::byte` stringification support (#1686) * Add `std::byte` support 1. Add byte detection in Catch config 2. Add a `std::byte`specialisation for Catch2::Detail::StringMaker
2019-07-13 14:47:56 +02:00
${SELF_TEST_DIR}/UsageTests/ToStringByte.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/ToStringChrono.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringGeneral.tests.cpp
Add to string for std::optional
2019-01-25 22:05:40 +01:00
${SELF_TEST_DIR}/UsageTests/ToStringOptional.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/ToStringPair.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringTuple.tests.cpp
Add StringMaker for std::variant, std::monostate (#1380) The StringMaker is off by default and can be enabled by a new macro `CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER`, to avoid increasing the footprint of stringification machinery by default.
2018-09-20 14:13:35 +02:00
${SELF_TEST_DIR}/UsageTests/ToStringVariant.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/ToStringVector.tests.cpp
${SELF_TEST_DIR}/UsageTests/ToStringWhich.tests.cpp
${SELF_TEST_DIR}/UsageTests/Tricky.tests.cpp
${SELF_TEST_DIR}/UsageTests/VariadicMacros.tests.cpp
Add generic Contains matcher It matches a range iff the range contains a specific element, or an element in the range matches the provided matcher.
2020-03-02 14:25:43 +01:00
${SELF_TEST_DIR}/UsageTests/MatchersRanges.tests.cpp
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
${SELF_TEST_DIR}/UsageTests/Matchers.tests.cpp
)
# Specify the headers, too, so CLion recognises them as project files
set(HEADERS
${TOP_LEVEL_HEADERS}
${EXTERNAL_HEADERS}
${INTERNAL_HEADERS}
${REPORTER_HEADERS}
Small CMakeLists cleanup
2019-12-18 17:28:19 +01:00
${BENCHMARK_HEADERS}
Move the async-using parts of benchmarking into a .cpp file This keeps it out of the main include path when benchmarking is enabled, somewhat reducing the compilation-time penalty. Also moved some other functions into the .cpp file, especially helpers that could be given internal linkage, and concretized some iterator-templated code that only ever used `std::vector<double>::iterator`.
2019-06-15 11:14:24 +02:00
${BENCHMARK_SOURCES}
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
# Provide some groupings for IDEs
Unit tests build and run (todo: projects, includes, extras)
2019-11-30 20:45:34 +01:00
#SOURCE_GROUP("benchmark" FILES ${BENCHMARK_HEADERS} ${BENCHMARK_SOURCES})
#SOURCE_GROUP("Tests" FILES ${TEST_SOURCES})
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
include(CTest)
WIP: CMake changes to build static lib + tests separately
2019-11-30 17:41:41 +01:00
add_executable(SelfTest ${TEST_SOURCES})
Remove catch_default_main.hpp There are two reasons for this: 1) It is highly unlikely that someone has use for this header, which has no customization points and only provides simplest possible main, and cannot link the static library which also provides a default main implementation. 2) It being a header was causing extra complications with the convenience headers, and our checking script. This would either require special handling in the checking script, or would break user's of the main convenience header. All in all, it is simpler and better in the long term to remove it, than to fix its problems.
2020-05-07 16:24:05 +02:00
target_link_libraries(SelfTest PRIVATE Catch2WithMain)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
if (CATCH_ENABLE_COVERAGE)
set(ENABLE_COVERAGE ON CACHE BOOL "Enable coverage build." FORCE)
find_package(codecov)
add_coverage(SelfTest)
list(APPEND LCOV_REMOVE_PATTERNS "'/usr/*'")
coverage_evaluate()
endif()
# configure unit tests via CTest
Run unit tests in random order
2020-05-03 09:49:59 +02:00
add_test(NAME RunTests COMMAND $<TARGET_FILE:SelfTest> --order rand --rng-seed time)
Add test for omitting the 'Filters' heading when there were none
2019-04-16 23:49:22 +02:00
set_tests_properties(RunTests PROPERTIES
FAIL_REGULAR_EXPRESSION "Filters:"
Remove the separately compiled test for Benchmarking Macros Thanks to the changes to compilation model, the tests for benchmarking macros have been made part of the normal test run. This means that the only purpose these separately compiled tests served was to waste CI time.
2020-02-18 13:48:34 +01:00
COST 60
Add test for omitting the 'Filters' heading when there were none
2019-04-16 23:49:22 +02:00
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
# Because CTest does not allow us to check both return code _and_ expected
# output in one test, we run these commands twice. First time we check
# the output, the second time we check the exit code.
add_test(NAME List::Tests::Output COMMAND $<TARGET_FILE:SelfTest> --list-tests --verbosity high)
set_tests_properties(List::Tests::Output PROPERTIES
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
PASS_REGULAR_EXPRESSION "[0-9]+ test cases"
FAIL_REGULAR_EXPRESSION "Hidden Test"
)
Special case --list-tests --verbosity quiet The new output (mostly) follows the old `--list-test-names-only` format, with the exception of no longer supporting line output for `--verbosity high`. Fixes #2051
2020-10-20 15:09:48 +02:00
# This should be equivalent to the old --list-test-names-only and be usable
# with --input-file.
add_test(NAME List::Tests::Quiet COMMAND $<TARGET_FILE:SelfTest> --list-tests --verbosity quiet)
# Sadly we cannot ask for start-of-line and end-of-line in a ctest regex,
# so we fail if we see space/tab at the start...
set_tests_properties(List::Tests::Quiet PROPERTIES
PASS_REGULAR_EXPRESSION "\"#1905 -- test spec parser properly clears internal state between compound tests\"[\r\n]"
FAIL_REGULAR_EXPRESSION "[ \t]\"#1905 -- test spec parser properly clears internal state between compound tests\""
)
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
add_test(NAME List::Tests::ExitCode COMMAND $<TARGET_FILE:SelfTest> --list-tests --verbosity high)
Specialize XML reporter's --list output
2019-06-22 15:31:11 +02:00
add_test(NAME List::Tests::XmlOutput COMMAND $<TARGET_FILE:SelfTest> --list-tests --verbosity high -r xml)
set_tests_properties(List::Tests::XmlOutput PROPERTIES
PASS_REGULAR_EXPRESSION "<Line>[0-9]+</Line>"
FAIL_REGULAR_EXPRESSION "[0-9]+ test cases"
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
add_test(NAME List::Tags::Output COMMAND $<TARGET_FILE:SelfTest> --list-tags)
set_tests_properties(List::Tags::Output PROPERTIES
PASS_REGULAR_EXPRESSION "[0-9]+ tags"
Fix CTest regex error The desired behaviour was to match a literal "[.]", so the regex has to be escaped as "\\[\\.\\]" -- double backslashes, because it has to be escaped from CMake as well as from the regex engine.
2018-09-08 20:56:31 +02:00
FAIL_REGULAR_EXPRESSION "\\[\\.\\]")
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
add_test(NAME List::Tags::ExitCode COMMAND $<TARGET_FILE:SelfTest> --list-tags)
Specialize XML reporter's --list output
2019-06-22 15:31:11 +02:00
add_test(NAME List::Tags::XmlOutput COMMAND $<TARGET_FILE:SelfTest> --list-tags -r xml)
set_tests_properties(List::Tags::XmlOutput PROPERTIES
PASS_REGULAR_EXPRESSION "<Count>18</Count>"
FAIL_REGULAR_EXPRESSION "[0-9]+ tags"
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
add_test(NAME List::Reporters::Output COMMAND $<TARGET_FILE:SelfTest> --list-reporters)
set_tests_properties(List::Reporters::Output PROPERTIES PASS_REGULAR_EXPRESSION "Available reporters:")
add_test(NAME List::Reporters::ExitCode COMMAND $<TARGET_FILE:SelfTest> --list-reporters)
Specialize XML reporter's --list output
2019-06-22 15:31:11 +02:00
add_test(NAME List::Reporters::XmlOutput COMMAND $<TARGET_FILE:SelfTest> --list-reporters -r xml)
set_tests_properties(List::Reporters::XmlOutput PROPERTIES
PASS_REGULAR_EXPRESSION "<Name>compact</Name>"
FAIL_REGULAR_EXPRESSION "Available reporters:"
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Make --list-* exit code be 0 Previously it returned the sum of listed things because ???. This was completely useless and in many ways actively counterproductive because of the success/failure conventions around exit codes. Closes #1410
2019-06-16 16:12:47 +02:00
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
NoAssertions runs only 1 test case instead of all of them
2020-02-03 21:24:53 +01:00
add_test(NAME NoAssertions COMMAND $<TARGET_FILE:SelfTest> -w NoAssertions "An empty test with no assertions")
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
set_tests_properties(NoAssertions PROPERTIES PASS_REGULAR_EXPRESSION "No assertions in test case")
Improve reporting of unmatched filters (#1684) This PR ultimately does 3 things * Separately tracks matched tests per each filter part (that is, a set of filters separated by an OR (`,`)), which allows Catch2 to report each of the alternative filters that don't match any tests. * Fixes `-w NoTests` to return non-zero in the process * Adds tests for `-w NoTests`.
2019-08-06 20:51:19 +02:00
add_test(NAME NoTest COMMAND $<TARGET_FILE:SelfTest> Tracker, "___nonexistent_test___")
set_tests_properties(NoTest PROPERTIES
PASS_REGULAR_EXPRESSION "No test cases matched '___nonexistent_test___'"
FAIL_REGULAR_EXPRESSION "No tests ran"
)
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
add_test(NAME WarnAboutNoTests COMMAND ${CMAKE_COMMAND} -P ${SELF_TEST_DIR}/WarnAboutNoTests.cmake $<TARGET_FILE:SelfTest>)
Improve reporting of unmatched filters (#1684) This PR ultimately does 3 things * Separately tracks matched tests per each filter part (that is, a set of filters separated by an OR (`,`)), which allows Catch2 to report each of the alternative filters that don't match any tests. * Fixes `-w NoTests` to return non-zero in the process * Adds tests for `-w NoTests`.
2019-08-06 20:51:19 +02:00
add_test(NAME UnmatchedOutputFilter COMMAND $<TARGET_FILE:SelfTest> [this-tag-does-not-exist] -w NoTests)
set_tests_properties(UnmatchedOutputFilter
PROPERTIES
PASS_REGULAR_EXPRESSION "No test cases matched '\\[this-tag-does-not-exist\\]'"
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Integrate tests for #1394 into our test suite
2019-01-15 23:13:16 +01:00
add_test(NAME FilteredSection-1 COMMAND $<TARGET_FILE:SelfTest> \#1394 -c RunSection)
set_tests_properties(FilteredSection-1 PROPERTIES FAIL_REGULAR_EXPRESSION "No tests ran")
add_test(NAME FilteredSection-2 COMMAND $<TARGET_FILE:SelfTest> \#1394\ nested -c NestedRunSection -c s1)
set_tests_properties(FilteredSection-2 PROPERTIES FAIL_REGULAR_EXPRESSION "No tests ran")
Fix potential infinite loops in generators combined with section filter The problem was that under specific circumstances, namely that none of their children progressed, `GeneratorTracker` will not progress. This was changed recently, to allow for code like this, where a `SECTION` follows a `GENERATE` at the same level: ```cpp SECTION("A") {} auto a = GENERATE(1, 2); SECTION("B") {} ``` However, this interacted badly with `SECTION` filters (`-c foo`), as they could deactivate all `SECTION`s below a generator, and thus stop it from progressing forever. This commit makes GeneratorTracker check whether there are any filters active, and if they are, it checks whether its section-children can ever run. Fixes #2025
2020-10-22 14:20:47 +02:00
add_test(
NAME
FilteredSection::GeneratorsDontCauseInfiniteLoop-1
COMMAND
$<TARGET_FILE:SelfTest> "#2025: original repro" -c "fov_0"
)
set_tests_properties(FilteredSection::GeneratorsDontCauseInfiniteLoop-1
PROPERTIES
PASS_REGULAR_EXPRESSION "inside with fov: 0" # This should happen
FAIL_REGULAR_EXPRESSION "inside with fov: 1" # This would mean there was no filtering
)
# GENERATE between filtered sections (both are selected)
add_test(
NAME
FilteredSection::GeneratorsDontCauseInfiniteLoop-2
COMMAND
$<TARGET_FILE:SelfTest> "#2025: same-level sections"
-c "A"
-c "B"
)
set_tests_properties(FilteredSection::GeneratorsDontCauseInfiniteLoop-2
PROPERTIES
PASS_REGULAR_EXPRESSION "All tests passed \\(4 assertions in 1 test case\\)"
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
# AppVeyor has a Python 2.7 in path, but doesn't have .py files as autorunnable
Moved scripts/ to tools/scripts/
2019-12-06 11:53:31 +01:00
add_test(NAME ApprovalTests COMMAND ${PYTHON_EXECUTABLE} ${CATCH_DIR}/tools/scripts/approvalTests.py $<TARGET_FILE:SelfTest>)
Provide CTest a hint on test costs (and thus ordering) When running tests in parallel, CTest runs the tests in decreasing order of cost (time required), to get the largest speed up from parallelism. However, the initial cost estimates for all tests are 0, and they are only updated after a test run. This works on a dev machine, where the tests are ran over and over again, because eventually the estimates become quite precise, but CI always does a clean build with 0 estimates. Because we have 2 slow tests, we want them to run first to avoid losing parallelism. To do this, we provide them with a cost estimate manually.
2020-02-03 09:07:23 +01:00
set_tests_properties(ApprovalTests
PROPERTIES
FAIL_REGULAR_EXPRESSION "Results differed"
COST 120 # We know that this is either the most, or second most,
# expensive test in the test suite, so we give it high estimate for CI runs
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
Add integration test for the children completion check fix
2019-06-30 00:00:41 +02:00
add_test(NAME RegressionCheck-1670 COMMAND $<TARGET_FILE:SelfTest> "#1670 regression check" -c A -r compact)
set_tests_properties(RegressionCheck-1670 PROPERTIES PASS_REGULAR_EXPRESSION "Passed 1 test case with 2 assertions.")
Integrate tests for #1394 into our test suite
2019-01-15 23:13:16 +01:00
Add a test for Catch version output
2019-08-03 20:16:46 +02:00
add_test(NAME VersionCheck COMMAND $<TARGET_FILE:SelfTest> -h)
set_tests_properties(VersionCheck PROPERTIES PASS_REGULAR_EXPRESSION "Catch v${PROJECT_VERSION}")
Add test for --libidentify command line flag
2019-09-07 20:36:12 +02:00
add_test(NAME LibIdentityTest COMMAND $<TARGET_FILE:SelfTest> --libidentify)
set_tests_properties(LibIdentityTest PROPERTIES PASS_REGULAR_EXPRESSION "description: A Catch2 test executable")
Add test for --filename-as-tags
2019-09-07 20:43:14 +02:00
add_test(NAME FilenameAsTagsTest COMMAND $<TARGET_FILE:SelfTest> -\# --list-tags)
set_tests_properties(FilenameAsTagsTest PROPERTIES PASS_REGULAR_EXPRESSION "\\[#Approx.tests\\]")
Fix matching of non-lowercase filename tags The bug was caused by forgetting to lower-case the filename tag for matching against test spec. Fixes #2064
2020-10-22 16:17:34 +02:00
# Check that the filename tags can also be matched against (#2064)
add_test(NAME FilenameAsTagsMatching COMMAND $<TARGET_FILE:SelfTest> -\# --list-tags [\#Approx.tests])
Tighten the test for matching filenames-as-tags
2020-10-23 23:02:09 +02:00
set_tests_properties(FilenameAsTagsMatching
PROPERTIES
PASS_REGULAR_EXPRESSION "\\[#Approx.tests\\]"
# Avoids false positives by looking for start of line (newline) before the 0
FAIL_REGULAR_EXPRESSION "[\r\n]0 tag"
)
Fix matching of non-lowercase filename tags The bug was caused by forgetting to lower-case the filename tag for matching against test spec. Fixes #2064
2020-10-22 16:17:34 +02:00
Add tests for escaping special characters in CLI
2019-09-21 00:11:45 +02:00
add_test(NAME EscapeSpecialCharactersInTestNames COMMAND $<TARGET_FILE:SelfTest> "Test with special\\, characters \"in name")
set_tests_properties(EscapeSpecialCharactersInTestNames PROPERTIES PASS_REGULAR_EXPRESSION "1 assertion in 1 test case")
Hidden tests now require positive filter match to be selected This also required some refactoring of how the pattern matching works. This means that the concepts of include and exclude patterns are no longer unified, with exclusion patterns working as just negation of an inclusion patterns (which led to including hidden tags by default, as they did not match the exclusion), but rather both include and exclude patterns are handled separately. The new logic is that given a filter and a test case, the test case must match _all_ include patterns and _no_ exclude patterns to be included by the filter. Furthermore, if the test case is hidden, then the filter must have at least one include pattern for the test case to be used. Closes #1184
2019-06-22 20:26:03 +02:00
add_test(NAME NegativeSpecNoHiddenTests COMMAND $<TARGET_FILE:SelfTest> --list-tests ~[approval])
set_tests_properties(NegativeSpecNoHiddenTests PROPERTIES FAIL_REGULAR_EXPRESSION "\\[\\.\\]")
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
add_test(NAME TestsInFile::SimpleSpecs COMMAND $<TARGET_FILE:SelfTest> "-f ${SELF_TEST_DIR}/Misc/plain-old-tests.input")
Add more tests for reading test specs from file Related to #1770
2019-10-20 15:14:50 +02:00
set_tests_properties(TestsInFile::SimpleSpecs PROPERTIES PASS_REGULAR_EXPRESSION "6 assertions in 2 test cases")
Add tests for the -f/--input-file command line argument
2019-10-09 12:48:55 +02:00
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
add_test(NAME TestsInFile::EscapeSpecialCharacters COMMAND $<TARGET_FILE:SelfTest> "-f ${SELF_TEST_DIR}/Misc/special-characters-in-file.input")
Add more tests for reading test specs from file Related to #1770
2019-10-20 15:14:50 +02:00
set_tests_properties(TestsInFile::EscapeSpecialCharacters PROPERTIES PASS_REGULAR_EXPRESSION "1 assertion in 1 test case")
# CTest does not allow us to create an AND of required regular expressions,
# so we have to split the test into 2 parts and look for parts of the expected
# output separately.
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
add_test(NAME TestsInFile::InvalidTestNames-1 COMMAND $<TARGET_FILE:SelfTest> "-f ${SELF_TEST_DIR}/Misc/invalid-test-names.input")
Add more tests for reading test specs from file Related to #1770
2019-10-20 15:14:50 +02:00
set_tests_properties(TestsInFile::InvalidTestNames-1 PROPERTIES PASS_REGULAR_EXPRESSION "Invalid Filter: \"Test with special, characters in \\\\\" name\"")
Move tests from projects/ to tests/
2019-12-05 16:00:20 +01:00
add_test(NAME TestsInFile::InvalidTestNames-2 COMMAND $<TARGET_FILE:SelfTest> "-f ${SELF_TEST_DIR}/Misc/invalid-test-names.input")
Add more tests for reading test specs from file Related to #1770
2019-10-20 15:14:50 +02:00
set_tests_properties(TestsInFile::InvalidTestNames-2 PROPERTIES PASS_REGULAR_EXPRESSION "No tests ran")
Add tests for the -f/--input-file command line argument
2019-10-09 12:48:55 +02:00
Add test for tag aliases
2019-12-18 17:30:41 +01:00
add_test(NAME TagAlias COMMAND $<TARGET_FILE:SelfTest> [@tricky] --list-tests)
set_tests_properties(TagAlias PROPERTIES
PASS_REGULAR_EXPRESSION "[0-9]+ matching test cases"
FAIL_REGULAR_EXPRESSION "0 matching test cases"
)
Add a test for Catch version output
2019-08-03 20:16:46 +02:00
Add test for consistent random ordering -- Fixed up during cherrypicking
2020-04-10 23:25:13 +02:00
add_test(NAME RandomTestOrdering COMMAND ${PYTHON_EXECUTABLE}
${CATCH_DIR}/tests/TestScripts/testRandomOrder.py $<TARGET_FILE:SelfTest>)
Add script checking convenience header correctness
2020-05-06 14:57:55 +02:00
add_test(NAME CheckConvenienceHeaders
COMMAND
${PYTHON_EXECUTABLE} ${CATCH_DIR}/tools/scripts/checkConvenienceHeaders.py
)
Integrate CMake with `<catch2/catch.hpp>` include paths This also goes for pkg-config installed by our CMake installation. This includes * Updating CMake version on Travis * Adding a `Catch2` subfolder to the `single_include/` folder to provide this include path both _inside_ the repository, and _outside_. * Updated examples to build with the new paths * Other general CMake cleanup
2018-06-11 10:48:10 +02:00
if (CATCH_USE_VALGRIND)
add_test(NAME ValgrindRunTests COMMAND valgrind --leak-check=full --error-exitcode=1 $<TARGET_FILE:SelfTest>)
add_test(NAME ValgrindListTests COMMAND valgrind --leak-check=full --error-exitcode=1 $<TARGET_FILE:SelfTest> --list-tests --verbosity high)
set_tests_properties(ValgrindListTests PROPERTIES PASS_REGULAR_EXPRESSION "definitely lost: 0 bytes in 0 blocks")
add_test(NAME ValgrindListTags COMMAND valgrind --leak-check=full --error-exitcode=1 $<TARGET_FILE:SelfTest> --list-tags)
set_tests_properties(ValgrindListTags PROPERTIES PASS_REGULAR_EXPRESSION "definitely lost: 0 bytes in 0 blocks")
endif()
Add development build option to CMake and enable it on CI Development build enables warnings and and `Werror` or equivalent.
2020-02-21 21:15:45 +01:00
list(APPEND CATCH_WARNING_TARGETS SelfTest)
set(CATCH_WARNING_TARGETS ${CATCH_WARNING_TARGETS} PARENT_SCOPE)
Reference in New Issue Copy Permalink