3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-22 13:26:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

Provide CTest a hint on test costs (and thus ordering)

Browse Source 
When running tests in parallel, CTest runs the tests in decreasing
order of cost (time required), to get the largest speed up from
parallelism. However, the initial cost estimates for all tests are
0, and they are only updated after a test run. This works on a dev
machine, where the tests are ran over and over again, because
eventually the estimates become quite precise, but CI always does
a clean build with 0 estimates.

Because we have 2 slow tests, we want them to run first to avoid
losing parallelism. To do this, we provide them with a cost estimate
manually.
This commit is contained in:
Martin Hořeňovský 2020-02-03 09:07:23 +01:00
parent b86ab20154
commit a3ffc20f57
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
2 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
tests/CMakeLists.txt
View File

@ -187,7 +187,12 @@ set_tests_properties(FilteredSection-2 PROPERTIES FAIL_REGULAR_EXPRESSION "No te
# AppVeyor has a Python 2.7 in path, but doesn't have .py files as autorunnable # AppVeyor has a Python 2.7 in path, but doesn't have .py files as autorunnable
add_test(NAME ApprovalTests COMMAND ${PYTHON_EXECUTABLE} ${CATCH_DIR}/tools/scripts/approvalTests.py $<TARGET_FILE:SelfTest>) add_test(NAME ApprovalTests COMMAND ${PYTHON_EXECUTABLE} ${CATCH_DIR}/tools/scripts/approvalTests.py $<TARGET_FILE:SelfTest>)
set_tests_properties(ApprovalTests PROPERTIES FAIL_REGULAR_EXPRESSION "Results differed") set_tests_properties(ApprovalTests
PROPERTIES
FAIL_REGULAR_EXPRESSION "Results differed"
COST 120 # We know that this is either the most, or second most,
# expensive test in the test suite, so we give it high estimate for CI runs
)
add_test(NAME RegressionCheck-1670 COMMAND $<TARGET_FILE:SelfTest> "#1670 regression check" -c A -r compact) add_test(NAME RegressionCheck-1670 COMMAND $<TARGET_FILE:SelfTest> "#1670 regression check" -c A -r compact)
set_tests_properties(RegressionCheck-1670 PROPERTIES PASS_REGULAR_EXPRESSION "Passed 1 test case with 2 assertions.") set_tests_properties(RegressionCheck-1670 PROPERTIES PASS_REGULAR_EXPRESSION "Passed 1 test case with 2 assertions.")

2
tests/ExtraTests/CMakeLists.txt
View File

@ -128,6 +128,8 @@ set_tests_properties(
BenchmarkingMacros BenchmarkingMacros
PROPERTIES PROPERTIES
PASS_REGULAR_EXPRESSION "benchmark name samples iterations estimated" PASS_REGULAR_EXPRESSION "benchmark name samples iterations estimated"
COST 80 # We know that this is either the most, or second most,
# expensive test in the test suite, so we give it high estimate for CI runs
) )
# This test touches windows.h, so it should only be compiled under msvc # This test touches windows.h, so it should only be compiled under msvc