3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-22 13:26:10 +01:00
Code Issues Packages Projects Releases Wiki Activity
506276c592
catch2/tests/SelfTest/Baselines/junit.sw.approved.txt

2306 lines
172 KiB
Plaintext
Raw Normal View History

XmlWriter reverts to XML 1.0. Character encodings that are not valid in XML 1.0 are instead written using C-style escapes
2017-02-06 17:00:05 +01:00
<?xml version="1.0" encoding="UTF-8"?>
Fix JUnit reporter output so that it conforms to JUnit schema The `properties` can only go under the `testsuite` tag, not under the `testsuites` tag. Fixes #1598
2019-04-11 13:04:54 +02:00
<testsuitesloose text artifact
>
Provide overloads for {Unordered}RangeEquals taking a std::initializer_list This allows writing something like const auto v = calculateSomeVectorOfInts(); CHECK_THAT(v, RangeEquals({1, 2, 3})); Fixes #2915.
2024-10-12 17:18:59 +02:00
<testsuite name="<exe-name>" errors="17" failures="130" skipped="12" tests="2281" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
Fix JUnit reporter output so that it conforms to JUnit schema The `properties` can only go under the `testsuite` tag, not under the `testsuites` tag. Fixes #1598
2019-04-11 13:04:54 +02:00
<properties>
<property name="random-seed" value="1"/>
Change reporters to report filters in round-trippable format
2022-10-27 20:43:00 +02:00
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
Fix JUnit reporter output so that it conforms to JUnit schema The `properties` can only go under the `testsuite` tag, not under the `testsuites` tag. Fixes #1598
2019-04-11 13:04:54 +02:00
</properties>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="# A test name that starts with a #" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1027: Bitfields can be captured" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1147" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1175 - Hidden Test" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1238" time="{duration}" status="run"/>
<testcase classname="<exe-name>.(Fixture_1245&lt;int, int>)" name="#1245" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="#1319: Sections can have description (even if it is not saved" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#1319: Sections can have description (even if it is not saved/SectionName" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1403" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1455 - INFO and WARN can start with a linebreak" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1514: stderr/stdout is not captured in tests aborted by an exception" time="{duration}" status="run">
Add tests for #1514 If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2019-03-02 21:22:41 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Add tests for #1514 If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2019-03-02 21:22:41 +01:00
1514
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Tricky.tests.cpp:<line number>
Add tests for #1514 If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2019-03-02 21:22:41 +01:00
</failure>
<system-out>
This would not be caught previously
</system-out>
<system-err>
Nor would this
</system-err>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#1548" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1905 -- test spec parser properly clears internal state between compound tests" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/Various parentheses" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/backslash in test name" time="{duration}" status="run"/>
Improved generator tracking * Successive executions of the same `GENERATE` macro (e.g. because of a for loop) no longer lead to multiple nested generators. * The same line can now contain multiple `GENERATE` macros without issues. Fixes #1913
2020-06-01 19:04:23 +02:00
<testcase classname="<exe-name>.global" name="#1913 - GENERATE inside a for loop should not keep recreating the generator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1913 - GENERATEs can share a line" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section" time="{duration}" status="run"/>
Modify generator tracking to allow GENERATEs between SECTIONs This means that code such as ```cpp TEST_CASE() { SECTION("first") { SUCCEED(); } auto _ = GENERATE(1, 2); SECTION("second") { SUCCEED(); } } ``` will run and report 3 assertions, 1 from section "first" and 2 from section "second". This also applies for greater and potentially more confusing nesting, but fundamentally it is up to the user to avoid overly complex and confusing nestings, just as with `SECTION`s. The old behaviour of `GENERATE` as first thing in a `TEST_CASE`, `GENERATE` not followed by a `SECTION`, etc etc should be unchanged. Closes #1938
2020-07-04 17:45:30 +02:00
<testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section/A" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section/B" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - Section followed by flat generate" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - Section followed by flat generate/A" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - flat generate" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - mixed sections and generates" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - mixed sections and generates/A" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - mixed sections and generates/B" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - nested generate" time="{duration}" status="run"/>
Fix copy paste error in 7-arg TEMPLATE_TEST_CASE_SIG implementation Closes #1954
2020-06-13 15:41:25 +02:00
<testcase classname="<exe-name>.global" name="#1954 - 7 arg template test case sig compiles - 1, 1, 1, 1, 1, 0, 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1954 - 7 arg template test case sig compiles - 5, 1, 1, 1, 1, 0, 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1954 - 7 arg template test case sig compiles - 5, 3, 1, 1, 1, 0, 0" time="{duration}" status="run"/>
Fix ulp distance calculation for numbers with different signs This is a simplification of the fix proposed in #2152, with the critical function split out so that it can be tested directly, without having to go through the ULP matcher. Closes #2152
2021-07-26 22:01:04 +02:00
<testcase classname="<exe-name>.global" name="#2152 - ULP checks between differently signed values were wrong - double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#2152 - ULP checks between differently signed values were wrong - float" time="{duration}" status="run"/>
Stop exceptions in generator constructors from aborting the binary Fixes #2615
2023-01-17 11:04:49 +01:00
<testcase classname="<exe-name>.global" name="#2615 - Throwing in constructor generator fails test case but does not abort" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<error type="TEST_CASE">
FAILED:
failure to init
at Generators.tests.cpp:<line number>
</error>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
expected exception
answer := 42
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/inside REQUIRE_NOTHROW" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Added approvals for “failed as expected” tests
2017-04-07 10:33:19 +02:00
<error message="thisThrows()" type="REQUIRE_NOTHROW">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE_NOTHROW( thisThrows() )
Added approvals for “failed as expected” tests
2017-04-07 10:33:19 +02:00
expected exception
answer := 42
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Added approvals for “failed as expected” tests
2017-04-07 10:33:19 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/inside REQUIRE_THROWS" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#809" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#833" time="{duration}" status="run"/>
Cleanups output normalization in ApprovalTests Removed: * NaN normalization * INFINITY normalization * errno normalization * Completely unused duration regex Tests using these macros should be tagged `[approvals]` so they are not run as part of approval tests. Also simplified regex for the test's executable filename, and hidden some tests relying on nullptr normalization.
2022-01-26 23:47:40 +01:00
<testcase classname="<exe-name>.global" name="#835 -- errno should not be touched by Catch2" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="f() == 0" type="CHECK">
FAILED:
CHECK( f() == 0 )
with expansion:
1 == 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Add tests for #835 Also add ErrnoGuard before `isatty` call, because apparently it can set errno to 25 (ENOTTY).
2017-03-06 22:07:33 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#872" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 4" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="'Not' checks that should fail" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure message="false != false" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( false != false )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
<failure message="true != true" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( true != true )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="!true" type="CHECK">
FAILED:
CHECK( !true )
with expansion:
false
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="!(true)" type="CHECK_FALSE">
FAILED:
CHECK_FALSE( true )
with expansion:
!true
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="!trueValue" type="CHECK">
FAILED:
CHECK( !trueValue )
with expansion:
false
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="!(trueValue)" type="CHECK_FALSE">
FAILED:
CHECK_FALSE( trueValue )
with expansion:
!true
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="!(1 == 1)" type="CHECK">
FAILED:
CHECK( !(1 == 1) )
with expansion:
false
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
<failure message="!(1 == 1)" type="CHECK_FALSE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK_FALSE( 1 == 1 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="'Not' checks that should succeed" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/compare to true" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/compare to false" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/negation" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/double negation" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/direct" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="3x3x3 ints" time="{duration}" status="run"/>
<testcase classname="<exe-name>.TestClass" name="A METHOD_AS_TEST_CASE based test run that fails" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="s == &quot;world&quot;" type="REQUIRE">
FAILED:
REQUIRE( s == "world" )
with expansion:
"hello" == "world"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.TestClass" name="A METHOD_AS_TEST_CASE based test run that succeeds" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo&lt;float>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>::m_a.size() == 1" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>::m_a.size() == 1 )
with expansion:
0 == 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Template tests: added TEMPLATE_PRODUCT_TEST_CASE support for generating test cases based on multiple template template types combined with template arguments for each of the template template types specified e.g. ``` TEMPLATE_PRODUCT_TEST_CASE("template product","[template]", (std::tuple, std::pair, std::map), ((int,float),(char,double),(int,char))) ``` will effectively create 9 test cases with types: std::tuple<int,float> std::tuple<char,double> std::tuple<int,char> std::pair<int,float> std::pair<char, double> std::pair<int,char> std::map<int,float> std::map<char,double> std::map<int,char> Tested type is accessible in test case body as TestType Unique name is created by appending ` - <index>` to test name since preprocessor has some limitations in recursions Closes #1454
2018-12-06 19:27:33 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo&lt;int>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>::m_a.size() == 1" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>::m_a.size() == 1 )
with expansion:
0 == 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Template tests: added TEMPLATE_PRODUCT_TEST_CASE support for generating test cases based on multiple template template types combined with template arguments for each of the template template types specified e.g. ``` TEMPLATE_PRODUCT_TEST_CASE("template product","[template]", (std::tuple, std::pair, std::map), ((int,float),(char,double),(int,char))) ``` will effectively create 9 test cases with types: std::tuple<int,float> std::tuple<char,double> std::tuple<int,char> std::pair<int,float> std::pair<char, double> std::pair<int,char> std::map<int,float> std::map<char,double> std::map<int,char> Tested type is accessible in test case body as TestType Unique name is created by appending ` - <index>` to test name since preprocessor has some limitations in recursions Closes #1454
2018-12-06 19:27:33 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector&lt;float>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>::m_a.size() == 1" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>::m_a.size() == 1 )
with expansion:
0 == 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Template tests: added TEMPLATE_PRODUCT_TEST_CASE support for generating test cases based on multiple template template types combined with template arguments for each of the template template types specified e.g. ``` TEMPLATE_PRODUCT_TEST_CASE("template product","[template]", (std::tuple, std::pair, std::map), ((int,float),(char,double),(int,char))) ``` will effectively create 9 test cases with types: std::tuple<int,float> std::tuple<char,double> std::tuple<int,char> std::pair<int,float> std::pair<char, double> std::pair<int,char> std::map<int,float> std::map<char,double> std::map<int,char> Tested type is accessible in test case body as TestType Unique name is created by appending ` - <index>` to test name since preprocessor has some limitations in recursions Closes #1454
2018-12-06 19:27:33 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector&lt;int>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>::m_a.size() == 1" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>::m_a.size() == 1 )
with expansion:
0 == 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Template tests: added TEMPLATE_PRODUCT_TEST_CASE support for generating test cases based on multiple template template types combined with template arguments for each of the template template types specified e.g. ``` TEMPLATE_PRODUCT_TEST_CASE("template product","[template]", (std::tuple, std::pair, std::map), ((int,float),(char,double),(int,char))) ``` will effectively create 9 test cases with types: std::tuple<int,float> std::tuple<char,double> std::tuple<int,char> std::pair<int,float> std::pair<char, double> std::pair<int,char> std::map<int,float> std::map<char,double> std::map<int,char> Tested type is accessible in test case body as TestType Unique name is created by appending ` - <index>` to test name since preprocessor has some limitations in recursions Closes #1454
2018-12-06 19:27:33 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo&lt;float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo&lt;int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector&lt;float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector&lt;int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2&lt;float, 6>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2 )
with expansion:
6 &lt; 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2&lt;int, 2>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2 )
with expansion:
2 &lt; 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array&lt;float, 6>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2 )
with expansion:
6 &lt; 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array&lt;int, 2>" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture_2&lt;TestType>{}.m_a.size() &lt; 2 )
with expansion:
2 &lt; 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2&lt;float,6>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2&lt;int,2>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array&lt;float,6>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture_2" name="A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array&lt;int,2>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that fails - double" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture&lt;TestType>::m_a == 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture&lt;TestType>::m_a == 2 )
with expansion:
1.0 == 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Add support for templated tests This adds support for templated tests and test methods via `TEMPLATE_TEST_CASE` and `TEMPLATE_TEST_CASE_METHOD` macros. These work mostly just like their regular counterparts*, but take an unlimited** number of types as their last arguments. * Unlike the plain `TEST_CASE*` macros, the `TEMPLATE*` variants require a tag string. ** In practice there is limit of about 300 types.
2018-11-08 07:26:39 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that fails - float" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture&lt;TestType>::m_a == 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture&lt;TestType>::m_a == 2 )
with expansion:
1.0f == 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Add support for templated tests This adds support for templated tests and test methods via `TEMPLATE_TEST_CASE` and `TEMPLATE_TEST_CASE_METHOD` macros. These work mostly just like their regular counterparts*, but take an unlimited** number of types as their last arguments. * Unlike the plain `TEST_CASE*` macros, the `TEMPLATE*` variants require a tag string. ** In practice there is limit of about 300 types.
2018-11-08 07:26:39 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that fails - int" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Template_Fixture&lt;TestType>::m_a == 2" type="REQUIRE">
FAILED:
REQUIRE( Template_Fixture&lt;TestType>::m_a == 2 )
with expansion:
1 == 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Add support for templated tests This adds support for templated tests and test methods via `TEMPLATE_TEST_CASE` and `TEMPLATE_TEST_CASE_METHOD` macros. These work mostly just like their regular counterparts*, but take an unlimited** number of types as their last arguments. * Unlike the plain `TEST_CASE*` macros, the `TEMPLATE*` variants require a tag string. ** In practice there is limit of about 300 types.
2018-11-08 07:26:39 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture" name="A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - int" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 1" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Nttp_Fixture&lt;V>::value == 0" type="REQUIRE">
FAILED:
REQUIRE( Nttp_Fixture&lt;V>::value == 0 )
with expansion:
1 == 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 3" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Nttp_Fixture&lt;V>::value == 0" type="REQUIRE">
FAILED:
REQUIRE( Nttp_Fixture&lt;V>::value == 0 )
with expansion:
3 == 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 6" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="Nttp_Fixture&lt;V>::value == 0" type="REQUIRE">
FAILED:
REQUIRE( Nttp_Fixture&lt;V>::value == 0 )
with expansion:
6 == 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Nttp support tests approvals
2019-05-23 19:29:18 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Nttp_Fixture" name="A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 6" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Fixture" name="A TEST_CASE_METHOD based test run that fails" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="m_a == 2" type="REQUIRE">
FAILED:
REQUIRE( m_a == 2 )
with expansion:
1 == 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Class.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Fixture" name="A TEST_CASE_METHOD based test run that succeeds" time="{duration}" status="run"/>
TEST_CASE_PERSISTENT_FIXTURE: A new fixture macro for allowing persistent fixtures throughout a TEST_CASE (#2885) This PR introduces a new `TEST_CASE` macro called `TEST_CASE_PERSISTENT_FIXTURE`. `TEST_CASE_PERSISTENT_FIXTURE` offers the same functionality as `TEST_CASE_METHOD` except for one difference. The object on which the test method is invoked is only created once for all invocations of the test case. The object is created just after the `testCaseStarting` event is broadcast and the object is destroyed just before the `testCaseEnding` event is broadcast. The main motivation for this new functionality is to allow `TEST_CASE`s to do expensive setup and teardown once per `TEST_CASE`, without having to resort to abusing event listeners or static function variables with manual initialization. Implements #1602 --------- Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2024-08-05 17:01:41 +02:00
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that fails/First partial run" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that fails/Second partial run" time="{duration}" status="run">
<failure message="m_a == 0" type="REQUIRE">
FAILED:
REQUIRE( m_a == 0 )
with expansion:
1 == 0
at Class.tests.cpp:<line number>
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds" time="{duration}" status="run"/>
TEST_CASE_PERSISTENT_FIXTURE: A new fixture macro for allowing persistent fixtures throughout a TEST_CASE (#2885) This PR introduces a new `TEST_CASE` macro called `TEST_CASE_PERSISTENT_FIXTURE`. `TEST_CASE_PERSISTENT_FIXTURE` offers the same functionality as `TEST_CASE_METHOD` except for one difference. The object on which the test method is invoked is only created once for all invocations of the test case. The object is created just after the `testCaseStarting` event is broadcast and the object is destroyed just before the `testCaseEnding` event is broadcast. The main motivation for this new functionality is to allow `TEST_CASE`s to do expensive setup and teardown once per `TEST_CASE`, without having to resort to abusing event listeners or static function variables with manual initialization. Implements #1602 --------- Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2024-08-05 17:01:41 +02:00
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds/First partial run" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds/Second partial run" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="A Template product test case - Foo&lt;float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case - Foo&lt;int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case - std::vector&lt;float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case - std::vector&lt;int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case with array signature - Bar&lt;float, 42>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case with array signature - Bar&lt;int, 9>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case with array signature - std::array&lt;float, 42>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case with array signature - std::array&lt;int, 9>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A comparison that uses literals instead of the normal constructor" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A couple of nested sections followed by a failure" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
to infinity and beyond
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="A couple of nested sections followed by a failure/Outer" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="A couple of nested sections followed by a failure/Outer/Inner" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A failing expression with a non streamable type is still captured" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="&amp;o1 == &amp;o2" type="CHECK">
FAILED:
CHECK( &amp;o1 == &amp;o2 )
with expansion:
0x<hex digits> == 0x<hex digits>
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Tricky.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="o1 == o2" type="CHECK">
FAILED:
CHECK( o1 == o2 )
with expansion:
{?} == {?}
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Tricky.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Absolute margin" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="An empty test with no assertions" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="An expression with side-effects should only be evaluated once" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="An unchecked exception reports the line of the last assertion" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error message="{Unknown expression after the reported line}">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
{Unknown expression after the reported line}
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
unexpected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Anonymous test case 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approx setters validate their arguments" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approx with exactly-representable margin" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate PI" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with different epsilons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with floats" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with ints" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with mixed numeric types" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Arbitrary predicate matcher" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Arbitrary predicate matcher/Function pointer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Arbitrary predicate matcher/Lambdas + different type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertion macros support bit operators and bool conversions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section/Another section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section/Another other section" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different argument ranges, same element type, default comparison" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different argument ranges, same element type, custom comparison" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different element type, custom comparisons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Can handle type that requires ADL-found free function begin and end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Initialization with move only types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Matching using matcher" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Simple, std-provided containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Type with empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Type requires ADL found empty free function" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE can deal with complex expressions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE can deal with complex expressions involving commas" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE parses string and character constants" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Capture and info messages" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Capture and info messages/Capture should stringify like assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Capture and info messages/Info should NOT stringify the way assertions do" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive" time="{duration}" status="run"/>
Add CaseInsensitiveEqualTo comparison type
2021-12-26 18:54:47 +01:00
<testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive/Degenerate cases" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive/Plain comparisons" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive" time="{duration}" status="run"/>
Add CaseInsensitiveEqualTo comparison type
2021-12-26 18:54:47 +01:00
<testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive/Degenerate cases" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive/Plain comparisons" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Character pretty printing" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Character pretty printing/Specifically escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Character pretty printing/General chars" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Character pretty printing/Low ASCII" time="{duration}" status="run"/>
Fix bug in TokenStream parser When presented with just '-' it would access the string at position [1]
2024-09-05 23:06:32 +02:00
<testcase classname="<exe-name>.global" name="Clara::Arg does not crash on incomplete input" time="{duration}" status="run"/>
Arg::parse accepts plain Args as argument
2020-11-07 17:51:50 +01:00
<testcase classname="<exe-name>.global" name="Clara::Arg supports single-arg parse the way Opt does" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas" time="{duration}" status="run"/>
Add support for multiply calling lambda parsers in Clara Previously a lambda parser in Clara could only be invoked once, even if it internally was ok with being invoked multiple times. With this change, a lambda parser can mark itself as `accept_many`, in which case it will be invoked multiple times if the appropriate flag was supplied multiple times by the user.
2021-10-27 18:57:56 +02:00
<testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas/Parsing fails on multiple options without accept_many" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas/Parsing succeeds on multiple options with accept_many" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="ColourGuard behaviour" time="{duration}" status="run"/>
Add tests for ColourGuard
2022-03-24 00:09:27 +01:00
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard is disengaged by default" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard is engaged by op&lt;&lt;" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard can be engaged explicitly" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Combining MatchAllOfGeneric does not nest" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining MatchAnyOfGeneric does not nest" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining MatchNotOfGeneric does not nest" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining concrete matchers does not use templated matchers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining only templated matchers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining templated and concrete matchers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Combining templated matchers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Commas in various macros are allowed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparing function pointers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparison ops" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparison with explicitly convertible types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparisons between ints where one side is computed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparisons between unsigned ints and negative signed ints match c++ standard behaviour" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparisons with int literals don't warn when mixing signed/ unsigned" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit" time="{duration}" status="run"/>
Add test for shortcircuiting behaviour of generic matcher combinators
2020-07-26 21:22:13 +02:00
<testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit/MatchAllOf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit/MatchAnyOf" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Composed matchers shortcircuit" time="{duration}" status="run"/>
Refactor how shortcircuiting of old style matchers is tested
2020-07-26 21:31:29 +02:00
<testcase classname="<exe-name>.global" name="Composed matchers shortcircuit/MatchAllOf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed matchers shortcircuit/MatchAnyOf" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Contains string matcher" time="{duration}" status="run">
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
<failure message="testStringForMatching(), ContainsSubstring( &quot;not there&quot;, Catch::CaseSensitive::No )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
CHECK_THAT( testStringForMatching(), ContainsSubstring( "not there", Catch::CaseSensitive::No ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" contains: "not there" (case
insensitive)
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add tests for case insensitive string matching
2017-11-13 15:46:33 +01:00
</failure>
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
<failure message="testStringForMatching(), ContainsSubstring( &quot;STRING&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
CHECK_THAT( testStringForMatching(), ContainsSubstring( "STRING" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" contains: "STRING"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Copy and then generate a range" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Copy and then generate a range/from var and iterators" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Copy and then generate a range/From a temporary container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Copy and then generate a range/Final validation" time="{duration}" status="run"/>
Add IStream::isStdOut
2022-02-21 23:48:15 +01:00
<testcase classname="<exe-name>.global" name="Cout stream properly declares it writes to stdout" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Custom exceptions can be translated when testing for nothrow" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<error message="throwCustom()" type="REQUIRE_NOTHROW">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE_NOTHROW( throwCustom() )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
custom exception - not std
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Custom exceptions can be translated when testing for throwing as something else" time="{duration}" status="run">
REQUIRE_THROWS_AS is now reported with expected type as well Also updated tests to reflect this change.
2017-02-09 12:41:16 +01:00
<error message="throwCustom(), std::exception" type="REQUIRE_THROWS_AS">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE_THROWS_AS( throwCustom(), std::exception )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
custom exception - not std
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Custom std-exceptions can be custom translated" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
custom std exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Default scale is invisible to comparison" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Directly creating an EnumInfo" time="{duration}" status="run"/>
Test & document SKIP in generator constructor Closes #1593
2023-05-31 15:12:23 +02:00
<testcase classname="<exe-name>.global" name="Empty generators can SKIP in constructor" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
This generator is empty
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
Add IStream::isStdOut
2022-02-21 23:48:15 +01:00
<testcase classname="<exe-name>.global" name="Empty stream name opens cout stream" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="EndsWith string matcher" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), EndsWith( &quot;Substring&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), EndsWith( "Substring" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" ends with: "Substring"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add tests for case insensitive string matching
2017-11-13 15:46:33 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), EndsWith( &quot;this&quot;, Catch::CaseSensitive::No )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), EndsWith( "this", Catch::CaseSensitive::No ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" ends with: "this" (case
insensitive)
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Fix wrong reference to REGISTER_ENUM This was renamed to CATCH_REGISTER_ENUM in 541f1ed1 (Only provide CATCH_REGISTER_ENUM, 2019-04-21).
2024-11-12 22:13:04 +01:00
<testcase classname="<exe-name>.global" name="Enums can quickly have stringification enabled using CATCH_REGISTER_ENUM" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Enums in namespaces can quickly have stringification enabled using CATCH_REGISTER_ENUM" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Epsilon only applies to Approx's value" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Equality checks that should fail" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven == 6" type="CHECK">
FAILED:
CHECK( data.int_seven == 6 )
with expansion:
7 == 6
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven == 8" type="CHECK">
FAILED:
CHECK( data.int_seven == 8 )
with expansion:
7 == 8
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven == 0" type="CHECK">
FAILED:
CHECK( data.int_seven == 0 )
with expansion:
7 == 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one == Approx( 9.11f )" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one == Approx( 9.11f ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f
==
Approx( 9.10999965667724609 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one == Approx( 9.0f )" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one == Approx( 9.0f ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f == Approx( 9.0 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one == Approx( 1 )" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one == Approx( 1 ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f == Approx( 1.0 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one == Approx( 0 )" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one == Approx( 0 ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f == Approx( 0.0 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.double_pi == Approx( 3.1415 )" type="CHECK">
FAILED:
CHECK( data.double_pi == Approx( 3.1415 ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
3.14159265350000005
==
Approx( 3.14150000000000018 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello == &quot;goodbye&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello == "goodbye" )
with expansion:
"hello" == "goodbye"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello == &quot;hell&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello == "hell" )
with expansion:
"hello" == "hell"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello == &quot;hello1&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello == "hello1" )
with expansion:
"hello" == "hello1"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello.size() == 6" type="CHECK">
FAILED:
CHECK( data.str_hello.size() == 6 )
with expansion:
5 == 6
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="x == Approx( 1.301 )" type="CHECK">
FAILED:
CHECK( x == Approx( 1.301 ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
1.30000000000000027
==
Approx( 1.30099999999999993 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Equality checks that should succeed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Equals" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Equals string matcher" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), Equals( &quot;this string contains 'ABC' as a substring&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), Equals( "this string contains 'ABC' as a substring" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" equals: "this string contains
'ABC' as a substring"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add tests for case insensitive string matching
2017-11-13 15:46:33 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), Equals( &quot;something else&quot;, Catch::CaseSensitive::No )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), Equals( "something else", Catch::CaseSensitive::No ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" equals: "something else" (case
insensitive)
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Exception as a value (e.g. in REQUIRE_THROWS_MATCHES) can be stringified" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception matchers that fail/No exception" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="doesNotThrow(), SpecialException, ExceptionMatcher{ 1 }" type="CHECK_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THROWS_MATCHES( doesNotThrow(), SpecialException, ExceptionMatcher{ 1 } )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="doesNotThrow(), SpecialException, ExceptionMatcher{ 1 }" type="REQUIRE_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
REQUIRE_THROWS_MATCHES( doesNotThrow(), SpecialException, ExceptionMatcher{ 1 } )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Exception matchers that fail/Type mismatch" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<error message="throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 }" type="CHECK_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THROWS_MATCHES( throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 } )
Update baselines
2017-06-06 15:07:57 +02:00
Unknown exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</error>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<error message="throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 }" type="REQUIRE_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
REQUIRE_THROWS_MATCHES( throwsAsInt( 1 ), SpecialException, ExceptionMatcher{ 1 } )
Update baselines
2017-06-06 15:07:57 +02:00
Unknown exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Exception matchers that fail/Contents are wrong" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="throwsSpecialException( 3 ), SpecialException, ExceptionMatcher{ 1 }" type="CHECK_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THROWS_MATCHES( throwsSpecialException( 3 ), SpecialException, ExceptionMatcher{ 1 } )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
SpecialException::what special exception has value of 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="throwsSpecialException( 4 ), SpecialException, ExceptionMatcher{ 1 }" type="REQUIRE_THROWS_MATCHES">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
REQUIRE_THROWS_MATCHES( throwsSpecialException( 4 ), SpecialException, ExceptionMatcher{ 1 } )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
SpecialException::what special exception has value of 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Exception matchers that succeed" time="{duration}" status="run"/>
Add MessageMatches matcher for exception (#2570)
2022-12-12 00:40:47 +01:00
<testcase classname="<exe-name>.global" name="Exception message can be matched" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Exception messages can be tested for" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/exact match" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/different case" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/wildcarded" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exceptions matchers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Expected exceptions that don't throw or unexpected exceptions fail the test" time="{duration}" status="run">
REQUIRE_THROWS_AS is now reported with expected type as well Also updated tests to reflect this change.
2017-02-09 12:41:16 +01:00
<error message="thisThrows(), std::string" type="CHECK_THROWS_AS">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK_THROWS_AS( thisThrows(), std::string )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
expected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</error>
REQUIRE_THROWS_AS is now reported with expected type as well Also updated tests to reflect this change.
2017-02-09 12:41:16 +01:00
<failure message="thisDoesntThrow(), std::domain_error" type="CHECK_THROWS_AS">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK_THROWS_AS( thisDoesntThrow(), std::domain_error )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
<error message="thisThrows()" type="CHECK_NOTHROW">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK_NOTHROW( thisThrows() )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
expected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="FAIL aborts the test" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
This is a failure
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="FAIL does not require an argument" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Added FAIL_CHECK Works like FAIL, but does not abort test. As proposed in #765
2017-03-08 16:40:20 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="FAIL_CHECK does not abort the test" time="{duration}" status="run">
Added FAIL_CHECK Works like FAIL, but does not abort test. As proposed in #765
2017-03-08 16:40:20 +01:00
<failure type="FAIL_CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Added FAIL_CHECK Works like FAIL, but does not abort test. As proposed in #765
2017-03-08 16:40:20 +01:00
This is a failure
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Factorials are computed" time="{duration}" status="run"/>
Add test for empty result of filter generator
2023-02-01 18:27:41 +01:00
<testcase classname="<exe-name>.global" name="Filter generator throws exception for empty generator" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Floating point matchers: double" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Relative" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Relative/Some subnormal values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Margin" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/ULPs" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Composed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Constructor validation" time="{duration}" status="run"/>
Add IsNaN matcher
2023-02-26 00:14:32 +01:00
<testcase classname="<exe-name>.global" name="Floating point matchers: double/IsNaN" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Floating point matchers: float" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Relative" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Relative/Some subnormal values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Margin" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/ULPs" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Composed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Constructor validation" time="{duration}" status="run"/>
Add IsNaN matcher
2023-02-26 00:14:32 +01:00
<testcase classname="<exe-name>.global" name="Floating point matchers: float/IsNaN" time="{duration}" status="run"/>
Add test showing literals and complex generators in one GENERATE
2023-12-10 16:38:48 +01:00
<testcase classname="<exe-name>.global" name="GENERATE can combine literals and generators" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate/Basic usage" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate/Throws if there are no matching values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Shortening a range" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Same type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Different type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Different deduced type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Repeating a generator" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Number of elements in source is divisible by chunk size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Number of elements in source is not divisible by chunk size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Chunk size of zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Throws on too small generators" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators -- simple" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators -- simple/one" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- simple/two" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Single value" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Preset values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Generator combinator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Explicitly typed generator sequence" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator" time="{duration}" status="run"/>
Add test for filtering out multiple initial values in filter gen
2021-10-28 11:26:34 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Simple filtering" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Filter out multiple elements at the start and end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Throws on construction if it can't get initial element" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Take generator" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Take generator/Take less" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Take generator/Take more" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Map with explicit return type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Map with deduced return type" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Repeat" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Repeat/Singular repeat" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Repeat/Actual repeat" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive auto step" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive auto step/Integer" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative auto step" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative auto step/Integer" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Slightly under end" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Slightly under end" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Slightly under end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Greater-than inequalities with different epsilons" time="{duration}" status="run"/>
Refactor tests
2022-03-31 22:30:38 +02:00
<testcase classname="<exe-name>.global" name="Hashers with different seed produce different hash with same test case" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashers with same seed produce same hash" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result" time="{duration}" status="run"/>
Refactor tests
2022-03-31 22:30:38 +02:00
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different test name" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different classname" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing test case produces same hash across multiple calls" time="{duration}" status="run"/>
Add test for multiple streaming parts in UNSCOPED_INFO
2023-09-17 10:44:31 +02:00
<testcase classname="<exe-name>.global" name="INFO and UNSCOPED_INFO can stream multiple arguments" time="{duration}" status="run">
<failure type="FAIL">
FAILED:
Show infos!
This info has multiple parts.
This unscoped info has multiple parts.
at Message.tests.cpp:<line number>
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="INFO and WARN do not abort tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="INFO gets logged on failure" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="a == 1" type="REQUIRE">
FAILED:
REQUIRE( a == 1 )
with expansion:
2 == 1
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
this message should be logged
so should this
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="INFO gets logged on failure, even if captured before successful assertions" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="a == 1" type="CHECK">
FAILED:
CHECK( a == 1 )
with expansion:
2 == 1
Update baselines
2017-06-06 15:07:57 +02:00
this message may be logged later
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
this message should be logged
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="a == 0" type="CHECK">
FAILED:
CHECK( a == 0 )
with expansion:
2 == 0
Update baselines
2017-06-06 15:07:57 +02:00
this message may be logged later
this message should be logged
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
and this, but later
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Rebased approvals with info loop change
2017-08-10 18:00:22 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="INFO is reset for each loop" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="i &lt; 10" type="REQUIRE">
FAILED:
REQUIRE( i &lt; 10 )
with expansion:
10 &lt; 10
Rebased approvals with info loop change
2017-08-10 18:00:22 +02:00
current counter 10
i := 10
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Assert Info reset need to also reset result disposition to normal to handle uncaught exception correctly (#2723) * AssertionEnd does not reset the assertion info yet. That is done after populateReaction. And reset assertion info would also reset the result disposition to normal, so that any uncaught exception would be reported as failure * Approving test output changes due to added unit tests * Unit tests to throw std::runtime_error instead of std::exception * Add a unit test to test incomplete assertion handler --------- Co-authored-by: Ross <ross.tang@gfo-x.com>
2023-08-07 22:07:31 +02:00
<testcase classname="<exe-name>.global" name="Incomplete AssertionHandler" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<error message="Dummy" type="REQUIRE">
FAILED:
REQUIRE( Dummy )
Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE
at AssertionHandler.tests.cpp:<line number>
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Inequality checks that should fail" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven != 7" type="CHECK">
FAILED:
CHECK( data.int_seven != 7 )
with expansion:
7 != 7
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one != Approx( 9.1f )" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one != Approx( 9.1f ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f
!=
Approx( 9.10000038146972656 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.double_pi != Approx( 3.1415926535 )" type="CHECK">
FAILED:
CHECK( data.double_pi != Approx( 3.1415926535 ) )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
3.14159265350000005
!=
Approx( 3.14159265350000005 )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello != &quot;hello&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello != "hello" )
with expansion:
"hello" != "hello"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello.size() != 5" type="CHECK">
FAILED:
CHECK( data.str_hello.size() != 5 )
with expansion:
5 != 5
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Inequality checks that should succeed" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="JsonWriter" time="{duration}" status="run"/>
Add JSON reporter (#2706) Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-11-14 22:52:15 +01:00
<testcase classname="<exe-name>.global" name="JsonWriter/Newly constructed JsonWriter does nothing" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeObject will create an empty pair of braces" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeObject with key will create an object to write the value" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/nesting objects" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeArray will create an empty pair of braces" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeArray creates array to write the values to" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Moved from JsonObjectWriter shall not insert superfluous brace" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Moved from JsonArrayWriter shall not insert superfluous bracket" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Custom class shall be quoted" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly" time="{duration}" status="run"/>
Fix special character escaping in JsonWriter
2023-11-14 23:35:22 +01:00
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Quote in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Backslash in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Forward slash in a string is **not** escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Backspace in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Formfeed in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/linefeed in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/carriage return in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/tab in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/combination of characters is escaped" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Lambdas in assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Less-than inequalities with different epsilons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ManuallyRegistered" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Matchers can be (AllOf) composed with the &amp;&amp; operator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Matchers can be (AnyOf) composed with the || operator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Matchers can be composed with both &amp;&amp; and ||" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Matchers can be composed with both &amp;&amp; and || - failing" time="{duration}" status="run">
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
<failure message="testStringForMatching(), ( ContainsSubstring( &quot;string&quot; ) || ContainsSubstring( &quot;different&quot; ) ) &amp;&amp; ContainsSubstring( &quot;random&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
CHECK_THAT( testStringForMatching(), ( ContainsSubstring( "string" ) || ContainsSubstring( "different" ) ) &amp;&amp; ContainsSubstring( "random" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" ( ( contains: "string" or
contains: "different" ) and contains: "random" )
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Matchers can be negated (Not) with the ! operator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Matchers can be negated (Not) with the ! operator - failing" time="{duration}" status="run">
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
<failure message="testStringForMatching(), !ContainsSubstring( &quot;substring&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
CHECK_THAT( testStringForMatching(), !ContainsSubstring( "substring" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" not contains: "substring"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
</testcase>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1/A" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
</testcase>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2/A" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
</testcase>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1/B" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
</testcase>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2/B" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Mismatching exception messages failing the test" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="thisThrows(), &quot;should fail&quot;" type="REQUIRE_THROWS_WITH">
FAILED:
REQUIRE_THROWS_WITH( thisThrows(), "should fail" )
with expansion:
"expected exception" equals: "should fail"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Add more unit tests * Call order of listeners/reporters in multireporter * Adding listeners/reporters properly updates reporter preferences
2021-12-19 15:16:56 +01:00
<testcase classname="<exe-name>.global" name="Multireporter calls reporters and listeners in correct order" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Multireporter updates ReporterPreferences properly" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Multireporter updates ReporterPreferences properly/Adding listeners" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Multireporter updates ReporterPreferences properly/Adding reporters" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Nested generators and captured variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Nice descriptive name" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Non-std exceptions can be translated" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
custom exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Objects that evaluated in boolean contexts can be checked" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Optionally static assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Ordering comparison checks that should fail" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven > 7" type="CHECK">
FAILED:
CHECK( data.int_seven > 7 )
with expansion:
7 > 7
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven &lt; 7" type="CHECK">
FAILED:
CHECK( data.int_seven &lt; 7 )
with expansion:
7 &lt; 7
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven > 8" type="CHECK">
FAILED:
CHECK( data.int_seven > 8 )
with expansion:
7 > 8
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven &lt; 6" type="CHECK">
FAILED:
CHECK( data.int_seven &lt; 6 )
with expansion:
7 &lt; 6
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven &lt; 0" type="CHECK">
FAILED:
CHECK( data.int_seven &lt; 0 )
with expansion:
7 &lt; 0
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven &lt; -1" type="CHECK">
FAILED:
CHECK( data.int_seven &lt; -1 )
with expansion:
7 &lt; -1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven >= 8" type="CHECK">
FAILED:
CHECK( data.int_seven >= 8 )
with expansion:
7 >= 8
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.int_seven &lt;= 6" type="CHECK">
FAILED:
CHECK( data.int_seven &lt;= 6 )
with expansion:
7 &lt;= 6
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one &lt; 9" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one &lt; 9 )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f &lt; 9
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one > 10" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one > 10 )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f > 10
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.float_nine_point_one > 9.2" type="CHECK">
FAILED:
CHECK( data.float_nine_point_one > 9.2 )
with expansion:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
9.100000381f > 9.19999999999999929
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello > &quot;hello&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello > "hello" )
with expansion:
"hello" > "hello"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello &lt; &quot;hello&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello &lt; "hello" )
with expansion:
"hello" &lt; "hello"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello > &quot;hellp&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello > "hellp" )
with expansion:
"hello" > "hellp"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello > &quot;z&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello > "z" )
with expansion:
"hello" > "z"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello &lt; &quot;hellm&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello &lt; "hellm" )
with expansion:
"hello" &lt; "hellm"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello &lt; &quot;a&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello &lt; "a" )
with expansion:
"hello" &lt; "a"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello >= &quot;z&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello >= "z" )
with expansion:
"hello" >= "z"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="data.str_hello &lt;= &quot;a&quot;" type="CHECK">
FAILED:
CHECK( data.str_hello &lt;= "a" )
with expansion:
"hello" &lt;= "a"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Condition.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Ordering comparison checks that should succeed" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds/Default seeded" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds/Specific seed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Output from all sections is reported/one" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
Message from section one
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Output from all sections is reported/two" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
Message from section two
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Overloaded comma or address-of operators are not used" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Parse uints" time="{duration}" status="run"/>
Add parseUInt utility function There is an increasing number of places where Catch2 wants to parse strings into numbers, but being stuck in C++14 world, we do not have good stdlib facilities to do this (`strtoul` and `stoul` are both bad).
2022-10-20 15:06:26 +02:00
<testcase classname="<exe-name>.global" name="Parse uints/proper inputs" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parse uints/Bad inputs" time="{duration}" status="run"/>
Add more tests for various less common tag-related cases
2021-12-27 10:04:48 +01:00
<testcase classname="<exe-name>.global" name="Parsed tags are matched case insensitive" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags" time="{duration}" status="run"/>
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/shard-count" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Negative shard count reports error" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Zero shard count reports error" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/shard-index" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Negative shard index reports error" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Shard index 0 is accepted" time="{duration}" status="run"/>
Add more tests for various less common tag-related cases
2021-12-27 10:04:48 +01:00
<testcase classname="<exe-name>.global" name="Parsing tags with non-alphabetical characters is pass-through" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Parsing warnings" time="{duration}" status="run"/>
Allow enabling multiple warnings in one invocation
2021-12-18 20:48:43 +01:00
<testcase classname="<exe-name>.global" name="Parsing warnings/NoAssertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing warnings/NoTests is no longer supported" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing warnings/Combining multiple warnings" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Pointers can be compared to null" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set/Floats" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set/Double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Predicate matcher can accept const char*" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/empty args don't cause a crash" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/default - no arguments" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case using" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case exclusion using exclude:" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case exclusion using ~" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/console" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/xml" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/--reporter/junit" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters" time="{duration}" status="run"/>
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/-b" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/--break" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-a aborts after first failure" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-x 2 aborts after two failures" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-x must be numeric" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress/Accepted options" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress/invalid options are reported" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow/-e" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow/--nothrow" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/output filename" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/output filename/-o filename" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/output filename/--out" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/combinations" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/combinations/Single character flags can be combined" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour/without option" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour/auto" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour/yes" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour/no" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/use-colour/error" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/samples" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/confidence-interval" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/no-analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/warmup-time" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int, double, float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int, double>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int>" time="{duration}" status="run"/>
Add generateRandomSeed utility to generate randomness seed
2021-10-08 20:02:24 +02:00
<testcase classname="<exe-name>.global" name="Random seed generation accepts known methods" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Random seed generation reports unknown methods" time="{duration}" status="run"/>
Support sentinel-based ranges in default stringify (#2004)
2020-08-18 10:34:47 +02:00
<testcase classname="<exe-name>.global" name="Range type with sentinel" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Reconstruction should be based on stringification: #914" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="truthy(false)" type="CHECK">
FAILED:
CHECK( truthy(false) )
with expansion:
Hey, its truthy!
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Decomposition.tests.cpp:<line number>
Update baselines
2017-06-06 15:07:57 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Regex string matcher" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), Matches( &quot;this STRING contains 'abc' as a substring&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), Matches( "this STRING contains 'abc' as a substring" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" matches "this STRING contains
'abc' as a substring" case sensitively
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Provide a regex matcher against std::string Related to #1040
2017-11-13 15:35:31 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), Matches( &quot;contains 'abc' as a substring&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), Matches( "contains 'abc' as a substring" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" matches "contains 'abc' as a
substring" case sensitively
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Provide a regex matcher against std::string Related to #1040
2017-11-13 15:35:31 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), Matches( &quot;this string contains 'abc' as a&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), Matches( "this string contains 'abc' as a" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" matches "this string contains
'abc' as a" case sensitively
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Provide a regex matcher against std::string Related to #1040
2017-11-13 15:35:31 +01:00
</failure>
</testcase>
Error out reporter registration for reporter names with '::'
2022-01-09 19:01:14 +01:00
<testcase classname="<exe-name>.global" name="Registering reporter with '::' in name fails" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Regression test #1" time="{duration}" status="run"/>
--list-* flags write to target specified by the -o flag Also added tests for the default implementations of list* reporter helpers. Closes #2061
2021-01-26 17:53:59 +01:00
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream" time="{duration}" status="run"/>
Use case in names of default reporters
2021-11-10 23:32:01 +01:00
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/Automake reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/Automake reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/Automake reporter lists tests" time="{duration}" status="run"/>
--list-* flags write to target specified by the -o flag Also added tests for the default implementations of list* reporter helpers. Closes #2061
2021-01-26 17:53:59 +01:00
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/compact reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/compact reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/compact reporter lists tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/console reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/console reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/console reporter lists tests" time="{duration}" status="run"/>
Add JSON reporter (#2706) Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-11-14 22:52:15 +01:00
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JSON reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JSON reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JSON reporter lists tests" time="{duration}" status="run"/>
Use case in names of default reporters
2021-11-10 23:32:01 +01:00
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JUnit reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JUnit reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/JUnit reporter lists tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/SonarQube reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/SonarQube reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/SonarQube reporter lists tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TAP reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TAP reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TAP reporter lists tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TeamCity reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TeamCity reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/TeamCity reporter lists tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/XML reporter lists tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/XML reporter lists reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Reporter's write listings to provided stream/XML reporter lists tests" time="{duration}" status="run"/>
Fix bad indentation calculation in the console reporter The problem came from the console reporter trying to provide a fancy linebreaking (primarily for things like `SCENARIO` or the BDD macros), so that new lines start with extra indentation if the text being line broken starts as "{text}: ". The console reporter did not properly take into account cases where the ": " part would already be in a later line, in which case it would ask for non-sensical level of indentation (larger than single line length). We fixed this by also enforcing that the special indentation case only triggers if the ": " is found early enough in the line, so that we also avoid degenerate cases like this: ``` blablabla: F a n c y . . . ``` Fixes #2309
2021-10-25 15:10:35 +02:00
<testcase classname="<exe-name>.global" name="Reproducer for #2309 - a very long description past 80 chars (default console width) with a late colon : blablabla" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="SUCCEED counts as a test pass" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="SUCCEED does not require an argument" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.Fixture" name="Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Fixture" name="Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods/Given: No operations precede me" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.Fixture" name="Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods/Given: No operations precede me/When: We get the count" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Fixture" name="Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods/Given: No operations precede me/When: We get the count/Then: Subsequently values are higher" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing/Given: This stuff exists" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing/Given: This stuff exists/And given: And some assumption" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing/Given: This stuff exists/And given: And some assumption/When: I do this" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing/Given: This stuff exists/And given: And some assumption/When: I do this/Then: it should do this" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: Do that thing with the thing/Given: This stuff exists/And given: And some assumption/When: I do this/Then: it should do this/And: do that" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: This is a really long scenario name to see how the list command deals with wrapping" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: This is a really long scenario name to see how the list command deals with wrapping/Given: A section name that is so long that it cannot fit in a single console width" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Scenario: This is a really long scenario name to see how the list command deals with wrapping/Given: A section name that is so long that it cannot fit in a single console width/When: The test headers are printed as part of the normal running of the scenario" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: This is a really long scenario name to see how the list command deals with wrapping/Given: A section name that is so long that it cannot fit in a single console width/When: The test headers are printed as part of the normal running of the scenario/Then: The, deliberately very long and overly verbose (you see what I did there?) section names must wrap, along with an indent" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: it is made larger" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: it is made larger/Then: the size and capacity go up" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: it is made larger/Then: the size and capacity go up/And when: it is made smaller again" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: it is made larger/Then: the size and capacity go up/And when: it is made smaller again/Then: the size goes down but the capacity stays the same" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: we reserve more space" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Scenario: Vector resizing affects size and capacity/Given: an empty vector/When: we reserve more space/Then: The capacity is increased but the size remains the same" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Sends stuff to stdout and stderr" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<system-out>
A string sent directly to stdout
</system-out>
<system-err>
A string sent directly to stderr
Added clog test
2017-12-05 18:48:15 +01:00
A string sent to stderr via clog
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</system-err>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Some simple comparisons between doubles" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Standard output from all sections is reported" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Standard output from all sections is reported/one" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Standard output from all sections is reported/two" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<system-out>
Message from section one
Message from section two
</system-out>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StartsWith string matcher" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), StartsWith( &quot;This String&quot; )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), StartsWith( "This String" ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" starts with: "This String"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add tests for case insensitive string matching
2017-11-13 15:46:33 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="testStringForMatching(), StartsWith( &quot;string&quot;, Catch::CaseSensitive::No )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( testStringForMatching(), StartsWith( "string", Catch::CaseSensitive::No ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
"this string contains 'abc' as a substring" starts with: "string" (case
insensitive)
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Static arrays are convertible to string" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Static arrays are convertible to string/Single item" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Static arrays are convertible to string/Multiple" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Static arrays are convertible to string/Non-trivial inner items" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="String matchers" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="StringRef" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StringRef/Empty string" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/From string literal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/From sub-string" time="{duration}" status="run"/>
Add tests for StringRef's copy operations being shallow
2021-03-08 14:57:59 +01:00
<testcase classname="<exe-name>.global" name="StringRef/Copy construction is shallow" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Copy assignment is shallow" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="StringRef/Substrings" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StringRef/Substrings/zero-based substring" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/non-zero-based substring" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/Pointer values of full refs should match" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/Pointer values of substring refs should also match" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/Past the end substring" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/Substring off the end are trimmed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Substrings/substring start after the end is empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/Comparisons are deep" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="StringRef/from std::string" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StringRef/from std::string/implicitly constructed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/from std::string/explicitly constructed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/from std::string/assigned" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="StringRef/to std::string" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StringRef/to std::string/explicitly constructed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/to std::string/assigned" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/std::string += StringRef" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef/StringRef + StringRef" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="StringRef at compilation time" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="StringRef at compilation time/Simple constructors" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="StringRef at compilation time/UDL construction" time="{duration}" status="run"/>
Cleanup and optimize stringifying of string-like types More specifically, made the actual implementation of string-like type handling take argument as `Catch::StringRef`, instead of taking `std::string const&`. This means that string-like types that are not `std::string` no longer need to pay for an extra construction of `std::string` (including the potential allocation), before they can be stringified. The actual string stringification routine is now also better about reserving sufficient space.
2021-05-18 00:02:46 +02:00
<testcase classname="<exe-name>.global" name="Stringifying char arrays with statically known sizes - char" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Stringifying char arrays with statically known sizes - signed char" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Stringifying char arrays with statically known sizes - unsigned char" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Stringifying std::chrono::duration helpers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Stringifying std::chrono::duration with weird ratios" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Stringifying std::chrono::time_point&lt;system_clock>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tabs and newlines show in output" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="s1 == s2" type="CHECK">
FAILED:
CHECK( s1 == s2 )
with expansion:
"if ($b == 10) {
$a = 20;
}"
==
"if ($b == 10) {
$a = 20;
}
"
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Tag alias can be registered against tag patterns" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Tag alias can be registered against tag patterns/The same tag alias can only be registered once" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tag alias can be registered against tag patterns/Tag aliases must be of the form [@name]" time="{duration}" status="run"/>
Add more tests for various less common tag-related cases
2021-12-27 10:04:48 +01:00
<testcase classname="<exe-name>.global" name="Tags with spaces and non-alphanumerical characters are accepted" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.Template_Fixture" name="Template test case method with test types specified inside std::tuple - MyTypes - 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture" name="Template test case method with test types specified inside std::tuple - MyTypes - 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Template_Fixture" name="Template test case method with test types specified inside std::tuple - MyTypes - 2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside std::tuple - MyTypes - 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside std::tuple - MyTypes - 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Template test case with test types specified inside std::tuple - MyTypes - 2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - float/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - int/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::string/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTest: vectors can be sized and resized - std::tuple&lt;int,float>/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - (std::tuple&lt;int, float>), 6/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - float,4/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - int,5/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="TemplateTestSig: vectors can be sized and resized - std::string,15/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
Add more tests for various less common tag-related cases
2021-12-27 10:04:48 +01:00
<testcase classname="<exe-name>.global" name="Test case with identical tags keeps just one" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Test case with one argument" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Test enum bit values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Test with special, characters &quot;in name" time="{duration}" status="run"/>
Suppress failure of CHECKED_IF and CHECKED_ELSE (#2187) Resolves #1390 Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2021-05-10 21:42:47 +02:00
<testcase classname="<exe-name>.global" name="Testing checked-if" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Testing checked-if 2" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Suppress failure of CHECKED_IF and CHECKED_ELSE (#2187) Resolves #1390 Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2021-05-10 21:42:47 +02:00
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Suppress failure of CHECKED_IF and CHECKED_ELSE (#2187) Resolves #1390 Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2021-05-10 21:42:47 +02:00
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="Testing checked-if 3" time="{duration}" status="run">
Mark !mayfail tests as skipped in the JUnit reporter Should fix #2116
2021-06-07 19:53:34 +02:00
<skipped message="TEST_CASE tagged with !mayfail"/>
Suppress failure of CHECKED_IF and CHECKED_ELSE (#2187) Resolves #1390 Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2021-05-10 21:42:47 +02:00
<failure type="FAIL">
FAILED:
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Suppress failure of CHECKED_IF and CHECKED_ELSE (#2187) Resolves #1390 Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2021-05-10 21:42:47 +02:00
</failure>
</testcase>
Assert Info reset need to also reset result disposition to normal to handle uncaught exception correctly (#2723) * AssertionEnd does not reset the assertion info yet. That is done after populateReaction. And reset assertion info would also reset the result disposition to normal, so that any uncaught exception would be reported as failure * Approving test output changes due to added unit tests * Unit tests to throw std::runtime_error instead of std::exception * Add a unit test to test incomplete assertion handler --------- Co-authored-by: Ross <ross.tang@gfo-x.com>
2023-08-07 22:07:31 +02:00
<testcase classname="<exe-name>.global" name="Testing checked-if 4" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<error message="{Unknown expression after the reported line}">
FAILED:
{Unknown expression after the reported line}
Uncaught exception should fail!
at Misc.tests.cpp:<line number>
</error>
</testcase>
<testcase classname="<exe-name>.global" name="Testing checked-if 5" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<error message="{Unknown expression after the reported line}">
FAILED:
{Unknown expression after the reported line}
Uncaught exception should fail!
at Misc.tests.cpp:<line number>
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="The NO_FAIL macro reports a failure but does not fail the test" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="The default listing implementation write to provided stream" time="{duration}" status="run"/>
--list-* flags write to target specified by the -o flag Also added tests for the default implementations of list* reporter helpers. Closes #2061
2021-01-26 17:53:59 +01:00
<testcase classname="<exe-name>.global" name="The default listing implementation write to provided stream/Listing tags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="The default listing implementation write to provided stream/Listing reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="The default listing implementation write to provided stream/Listing tests" time="{duration}" status="run"/>
Add --list-listeners option
2022-05-10 20:00:36 +02:00
<testcase classname="<exe-name>.global" name="The default listing implementation write to provided stream/Listing listeners" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="This test 'should' fail but doesn't" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Thrown string literals are translated" time="{duration}" status="run">
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
For some reason someone is throwing a string literal!
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Tracker" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/successfully close one section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/fail one section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/fail one section/re-enter after failed section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/fail one section/re-enter after failed section and find next section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/successfully close one section, then find another" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/successfully close one section, then find another/Re-enter - skips S1 and enters S2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/successfully close one section, then find another/Re-enter - skips S1 and enters S2/Successfully close S2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/successfully close one section, then find another/Re-enter - skips S1 and enters S2/fail S2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Tracker/open a nested section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Trim strings" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Container conversions" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Container conversions/Two equal containers of different container types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Container conversions/Two equal containers of different container types (differ in array N)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Container conversions/Two equal containers of different container types and value types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Container conversions/Two equal containers, one random access, one not" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Value type" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Value type/Two equal containers of different value types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Value type/Two non-equal containers of different value types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Ranges with begin that needs ADL" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Custom predicate" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Type conversions of RangeEquals and similar/Custom predicate/Two equal non-empty containers (close enough)" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Unexpected exceptions can be translated" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Default StringMaker<FloatingPointType>::precision to max_digits10 `max_digits10` is the number of decimal digits that the type _can_ represent, in other words, the number of decimal digits needed to disambiguate any two floating point numbers when serialized to a string.
2024-04-30 16:43:05 +02:00
3.14000000000000012
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Upcasting special member functions" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Upcasting special member functions/Move constructor" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Upcasting special member functions/move assignment" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher/Basic usage" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher/Type requires ADL found begin and end" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AllMatch range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Basic usage" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Basic usage/All true evaluates to true" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Basic usage/Empty evaluates to true" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Basic usage/One false evaluates to false" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Basic usage/All false evaluates to false" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Contained type is convertible to bool" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Contained type is convertible to bool/All true evaluates to true" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Contained type is convertible to bool/One false evaluates to false" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Contained type is convertible to bool/All false evaluates to false" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AllTrue range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher/Basic usage" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher/Type requires ADL found begin and end" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AnyMatch range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Basic usage" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Basic usage/All true evaluates to true" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Basic usage/Empty evaluates to false" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Basic usage/One true evaluates to true" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Basic usage/All false evaluates to false" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Contained type is convertible to bool" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Contained type is convertible to bool/All true evaluates to true" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Contained type is convertible to bool/One true evaluates to true" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Contained type is convertible to bool/All false evaluates to false" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of AnyTrue range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher/Basic usage" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher/Type requires ADL found begin and end" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add more comprehensive tests for the quantifier matchers This includes * Testing both positive and negative path through the matchers * Testing them with types whose `begin` and `end` member functions require ADL * Testing them with types that return different types from `begin` and `end`
2020-12-26 23:09:51 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of NoneMatch range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Basic usage" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Basic usage/All true evaluates to false" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Basic usage/Empty evaluates to true" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Basic usage/One true evaluates to false" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Basic usage/All false evaluates to true" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Contained type is convertible to bool" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Contained type is convertible to bool/All true evaluates to false" time="{duration}" status="run"/>
Fix spelling
2023-04-25 20:14:27 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Contained type is convertible to bool/One true evaluates to false" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Contained type is convertible to bool/All false evaluates to true" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Shortcircuiting" time="{duration}" status="run"/>
Add AllTrue, AnyTrue, NoneTrue matchers
2021-11-14 17:05:31 +01:00
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Shortcircuiting/All are read" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of NoneTrue range matcher/Shortcircuiting/Short-circuited" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Empty container matches empty container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Empty container does not match non-empty container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Two equal 1-length non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Two equal-sized, equal, non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Two equal-sized, non-equal, non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Basic usage/Two non-equal-sized, non-empty containers (with same first elements)" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Custom predicate" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Custom predicate/Two equal non-empty containers (close enough)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Custom predicate/Two non-equal non-empty containers (close enough)" time="{duration}" status="run"/>
Fix RangeEquals matcher to handle iterator + sentinel ranges Also added tests for types that require ADL lookup for their `begin` and `end`.
2023-02-03 16:12:09 +01:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Ranges that need ADL begin/end" time="{duration}" status="run"/>
Provide overloads for {Unordered}RangeEquals taking a std::initializer_list This allows writing something like const auto v = calculateSomeVectorOfInts(); CHECK_THAT(v, RangeEquals({1, 2, 3})); Fixes #2915.
2024-10-12 17:18:59 +02:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Compare against std::initializer_list" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Check short-circuiting behaviour" time="{duration}" status="run"/>
Fix RangeEquals matcher to handle iterator + sentinel ranges Also added tests for types that require ADL lookup for their `begin` and `end`.
2023-02-03 16:12:09 +01:00
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Check short-circuiting behaviour/Check short-circuits on failure" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of RangeEquals range matcher/Check short-circuiting behaviour/All elements are checked on success" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Empty container matches empty container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Empty container does not match non-empty container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Two equal 1-length non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Two equal-sized, equal, non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Two equal-sized, non-equal, non-empty containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Basic usage/Two non-equal-sized, non-empty containers" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Custom predicate" time="{duration}" status="run"/>
Added ElementsAre and UnorderedElementsAre (#2377) Co-authored-by: Garz4 <fancygarz4@gmail.com> Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-22 00:33:04 +01:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Custom predicate/Two equal non-empty containers (close enough)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Custom predicate/Two non-equal non-empty containers (close enough)" time="{duration}" status="run"/>
Enable use of UnorderedRangeEquals with iterator+sentinel pairs
2023-02-06 15:34:32 +01:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Ranges that need ADL begin/end" time="{duration}" status="run"/>
Provide overloads for {Unordered}RangeEquals taking a std::initializer_list This allows writing something like const auto v = calculateSomeVectorOfInts(); CHECK_THAT(v, RangeEquals({1, 2, 3})); Fixes #2915.
2024-10-12 17:18:59 +02:00
<testcase classname="<exe-name>.global" name="Usage of UnorderedRangeEquals range matcher/Compare against std::initializer_list" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Usage of the SizeIs range matcher" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Usage of the SizeIs range matcher/Some with stdlib containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of the SizeIs range matcher/Type requires ADL found size free function" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Usage of the SizeIs range matcher/Type has size member" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Use a custom approx" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Variadic macros" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Variadic macros/Section with one argument" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher/Empty vector is roughly equal to an empty vector" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher/Vectors with elements" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher/Vectors with elements/A vector is approx equal to itself" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector Approx matcher/Vectors with elements/Different length" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector Approx matcher/Vectors with elements/Same length, different elements" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher -- failing" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher -- failing/Empty and non empty vectors are not approx equal" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="empty, Approx( t1 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( empty, Approx( t1 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ } is approx: { 1.0, 2.0 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added a vector approximate matcher
2019-01-14 15:31:09 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector Approx matcher -- failing/Just different vectors" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v1, Approx( v2 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v1, Approx( v2 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 2.0, 4.0, 6.0 } is approx: { 1.0, 3.0, 5.0 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added a vector approximate matcher
2019-01-14 15:31:09 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers/Contains (element)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector matchers/Contains (vector)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector matchers/Contains (element), composed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector matchers/Equals" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Vector matchers/UnorderedEquals" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers that fail" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers that fail/Contains (element)" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v, VectorContains( -1 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v, VectorContains( -1 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2, 3 } Contains: -1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="empty, VectorContains( 1 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( empty, VectorContains( 1 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ } Contains: 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers that fail/Contains (vector)" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="empty, Contains( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( empty, Contains( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ } Contains: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v, Contains( v2 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v, Contains( v2 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2, 3 } Contains: { 1, 2, 4 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers that fail/Equals" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v, Equals( v2 )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v, Equals( v2 ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2, 3 } Equals: { 1, 2 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v2, Equals( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v2, Equals( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2 } Equals: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="empty, Equals( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( empty, Equals( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ } Equals: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v, Equals( empty )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v, Equals( empty ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2, 3 } Equals: { }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add UnorderedEqualsMatcher for vectors Closes #1093
2017-12-07 17:07:25 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="Vector matchers that fail/UnorderedEquals" time="{duration}" status="run">
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="v, UnorderedEquals( empty )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( v, UnorderedEquals( empty ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 2, 3 } UnorderedEquals: { }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add UnorderedEqualsMatcher for vectors Closes #1093
2017-12-07 17:07:25 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="empty, UnorderedEquals( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( empty, UnorderedEquals( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ } UnorderedEquals: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add UnorderedEqualsMatcher for vectors Closes #1093
2017-12-07 17:07:25 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="permuted, UnorderedEquals( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( permuted, UnorderedEquals( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 1, 3 } UnorderedEquals: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Add UnorderedEqualsMatcher for vectors Closes #1093
2017-12-07 17:07:25 +01:00
</failure>
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
<failure message="permuted, UnorderedEquals( v )" type="CHECK_THAT">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Remove the ill-conceived compilation perf tests using real tests
2021-06-20 16:25:57 +02:00
CHECK_THAT( permuted, UnorderedEquals( v ) )
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
with expansion:
{ 3, 1 } UnorderedEquals: { 1, 2, 3 }
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Matchers.tests.cpp:<line number>
Added first vector matchers (Contains and Equals)
2017-02-21 17:05:04 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="When checked exceptions are thrown they can be expected or unexpected" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown directly they are always failures" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
unexpected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown during a CHECK the test should continue" time="{duration}" status="run">
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
<error message="thisThrows() == 0" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( thisThrows() == 0 )
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
expected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Fix for #270 (REQUIRE continues if test throws an exception). Added test case that reproduced the bug then applied the fix.
2014-04-12 20:07:24 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown during a REQUIRE the test should abort fail" time="{duration}" status="run">
Fix for #270 (REQUIRE continues if test throws an exception). Added test case that reproduced the bug then applied the fix.
2014-04-12 20:07:24 +02:00
<error message="thisThrows() == 0" type="REQUIRE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE( thisThrows() == 0 )
Fix for #270 (REQUIRE continues if test throws an exception). Added test case that reproduced the bug then applied the fix.
2014-04-12 20:07:24 +02:00
expected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Secondary fix for #270 (addresses same issue with CHECK)
2014-04-12 20:20:46 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown from functions they are always failures" time="{duration}" status="run">
Secondary fix for #270 (addresses same issue with CHECK)
2014-04-12 20:20:46 +02:00
<error message="thisThrows() == 0" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( thisThrows() == 0 )
Secondary fix for #270 (addresses same issue with CHECK)
2014-04-12 20:20:46 +02:00
expected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown from sections they are always failures/section name" time="{duration}" status="run">
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
approved tests that were correct the first time (previous approvals were erroneous)
2017-04-09 21:40:01 +02:00
unexpected exception
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</error>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="When unchecked exceptions are thrown, but caught, they do not affect the test" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="X/level/0/a" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="X/level/0/b" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="X/level/1/a" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="X/level/1/b" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="XmlEncode" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="XmlEncode/normal string" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/empty string" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with ampersand" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with less-than" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with greater-than" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with quotes" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (1)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="XmlEncode/string with control char (x7F)" time="{duration}" status="run"/>
Add explicit test for serialization of boolean attributes in XML
2021-05-30 20:02:02 +02:00
<testcase classname="<exe-name>.global" name="XmlWriter writes boolean attributes as true/false" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="a succeeding test can still be skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="analyse no analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="array&lt;int, N> -> toString" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="benchmark function call" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="benchmark function call/without chronometer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="benchmark function call/with chronometer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="boolean member" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="checkedElse" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="checkedElse, failing" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="testCheckedElse( false )" type="REQUIRE">
FAILED:
REQUIRE( testCheckedElse( false ) )
with expansion:
false
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="checkedIf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="checkedIf, failing" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="testCheckedIf( false )" type="REQUIRE">
FAILED:
REQUIRE( testCheckedIf( false ) )
with expansion:
false
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="classify_outliers" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="classify_outliers/none" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="classify_outliers/low severe" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="classify_outliers/low mild" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="classify_outliers/high mild" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="classify_outliers/high severe" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="classify_outliers/mixed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="comparisons between const int variables" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="comparisons between int variables" time="{duration}" status="run"/>
Fix ulp distance calculation for numbers with different signs This is a simplification of the fix proposed in #2152, with the critical function split out so that it can be tested directly, without having to go through the ULP matcher. Closes #2152
2021-07-26 22:01:04 +02:00
<testcase classname="<exe-name>.global" name="convertToBits" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="dynamic skipping works with generators" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 41
at Skip.tests.cpp:<line number>
</skipped>
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
Disallow empty tags in test case specification
2021-05-28 20:49:05 +02:00
<testcase classname="<exe-name>.global" name="empty tags are not allowed" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="erfc_inv" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="estimate_clock_resolution" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="even more nested SECTION tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/d (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/c/e (leaf)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="even more nested SECTION tests/f (leaf)" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="failed assertions before SKIP cause test case to fail" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure message="3 == 4" type="CHECK">
FAILED:
CHECK( 3 == 4 )
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing for some generator values causes entire test case to fail" time="{duration}" status="run">
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="failing in some unskipped sections causes entire test case to fail/not skipped" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
FAILED:
at Skip.tests.cpp:<line number>
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="first tag" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="has printf" time="{duration}" status="run"/>
Add support for multiply calling lambda parsers in Clara Previously a lambda parser in Clara could only be invoked once, even if it internally was ok with being invoked multiple times. With this change, a lambda parser can mark itself as `accept_many`, in which case it will be invoked multiple times if the appropriate flag was supplied multiple times by the user.
2021-10-27 18:57:56 +02:00
<testcase classname="<exe-name>.global" name="is_unary_function" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="just failure" time="{duration}" status="run">
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Previous info should not be seen
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="just failure after unscoped info" time="{duration}" status="run">
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
<failure type="FAIL">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
previous unscoped info SHOULD not be seen
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="just info" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="just unscoped info" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="long long" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="looped SECTION tests" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 0" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="b > a" type="CHECK">
FAILED:
CHECK( b > a )
with expansion:
0 > 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 1" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="b > a" type="CHECK">
FAILED:
CHECK( b > a )
with expansion:
1 > 1
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Deprecated description in SECTION (still accepts it, for now, but doesn't use it anywhere)
2018-06-25 20:04:29 +02:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 4" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 5" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 6" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 7" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 8" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped SECTION tests/b is currently: 9" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="looped tests" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[0] (1) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[1] (1) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[3] (3) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[4] (5) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[6] (13) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="( fib[i] % 2 ) == 0" type="CHECK">
FAILED:
CHECK( ( fib[i] % 2 ) == 0 )
with expansion:
1 == 0
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Testing if fib[7] (21) is even
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Add test for %debug stream name
2022-04-23 23:34:27 +02:00
<testcase classname="<exe-name>.global" name="makeStream recognizes %debug stream name" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="make_unique reimplementation" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="make_unique reimplementation/From lvalue copies" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="make_unique reimplementation/From rvalue moves" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="make_unique reimplementation/Variadic constructor" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="mean" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="measure" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="mix info, unscoped info and warning" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="more nested SECTION tests" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="more nested SECTION tests/equal" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="more nested SECTION tests/equal/doesn't equal" time="{duration}" status="run">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
<failure message="a == b" type="REQUIRE">
FAILED:
REQUIRE( a == b )
with expansion:
1 == 2
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="more nested SECTION tests/doesn't equal/less than" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="nested SECTION tests" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested SECTION tests/doesn't equal/not equal" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/A" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B2" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B2/B1" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B2/B" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="nested sections can be skipped dynamically at runtime/B" time="{duration}" status="run">
<system-out>
a!
b1!
!
</system-out>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="non streamable - with conv. op" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="non-copyable objects" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="normal_quantile" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="not allowed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="not prints unscoped info from previous failures" time="{duration}" status="run">
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
<failure message="false" type="REQUIRE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE( false )
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
this SHOULD be seen
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="null strings" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="null_ptr" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="pair&lt;pair&lt;int,const char *,pair&lt;std::string,int> > -> toString" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="parseEnums" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="parseEnums/No enums" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="parseEnums/One enum value" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="parseEnums/Multiple enum values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="pointer to class" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="print unscoped info if passing unscoped info is printed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="prints unscoped info on failure" time="{duration}" status="run">
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
<failure message="false" type="REQUIRE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE( false )
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
this SHOULD be seen
this SHOULD also be seen
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="prints unscoped info only for the first assertion" time="{duration}" status="run">
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
<failure message="false" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( false )
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
this SHOULD be seen only ONCE
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
</testcase>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="random SECTION tests" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="random SECTION tests/doesn't equal" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="random SECTION tests/not equal" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="replaceInPlace" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="replaceInPlace/replace single char" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/replace two chars" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/replace first char" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/replace last char" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/replace all chars" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/replace no chars" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="replaceInPlace/no replace in already-replaced string" time="{duration}" status="run"/>
less copies and allocations in replaceInPlace
2024-02-25 18:01:28 +01:00
<testcase classname="<exe-name>.global" name="replaceInPlace/no replace in already-replaced string/lengthening" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="replaceInPlace/no replace in already-replaced string/shortening" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="replaceInPlace/escape '" time="{duration}" status="run"/>
Add %stderr and %stdout as recognized magic stream names
2022-03-07 10:42:26 +01:00
<testcase classname="<exe-name>.global" name="request an unknown %-starting stream fails" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="resolution" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, chronometer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="run_for_at_least, int" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="second tag" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/not skipped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/skipped" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
<testcase classname="<exe-name>.global" name="sections can be skipped dynamically at runtime/also not skipped" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="send a single char to INFO" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure message="false" type="REQUIRE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE( false )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
3
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Misc.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="sends information to INFO" time="{duration}" status="run">
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
<failure message="false" type="REQUIRE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
REQUIRE( false )
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
hi
i := 7
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="shortened hide tags are split apart" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="skipped tests can optionally provide a reason" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
skipping because answer = 43
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="splitString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stacks unscoped info in loops" time="{duration}" status="run">
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
<failure message="false" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( false )
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
Count 1 to 3...
1
2
3
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
</failure>
<failure message="false" type="CHECK">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
CHECK( false )
Add tests for unscoped info Update approval tests as new tests are added for messaging.
2019-02-01 16:36:35 +01:00
Count 4 to 6...
4
5
6
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Message.tests.cpp:<line number>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</failure>
</testcase>
Change startsWith(char) to take StringRef as argument
2021-09-27 14:52:44 +02:00
<testcase classname="<exe-name>.global" name="startsWith" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="std::map is convertible string" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="std::map is convertible string/empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::map is convertible string/single item" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::map is convertible string/several items" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::pair&lt;int,const std::string> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::pair&lt;int,std::string> -> toString" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="std::set is convertible string" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="std::set is convertible string/empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::set is convertible string/single item" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::set is convertible string/several items" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="std::vector&lt;std::pair&lt;std::string,int> > -> toString" time="{duration}" status="run"/>
Add %stderr and %stdout as recognized magic stream names
2022-03-07 10:42:26 +01:00
<testcase classname="<exe-name>.global" name="stdout and stderr streams have %-starting name" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="stringify ranges" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( has_maker )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( has_maker_and_operator )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( has_neither )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( has_operator )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( has_template_operator )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( vectors&lt;has_maker> )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( vectors&lt;has_maker_and_operator> )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="stringify( vectors&lt;has_operator> )" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="strlen3" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tables" time="{duration}" status="run"/>
Add test for dots in tag names
2021-05-12 21:43:14 +02:00
<testcase classname="<exe-name>.global" name="tags with dots in later positions are not parsed as hidden" time="{duration}" status="run"/>
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
<testcase classname="<exe-name>.global" name="tests can be skipped dynamically at runtime" time="{duration}" status="run">
<skipped type="SKIP">
SKIPPED
at Skip.tests.cpp:<line number>
</skipped>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="thrown std::strings are translated" time="{duration}" status="run">
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
<error type="TEST_CASE">
Improve JUnit failure report The JUnit report is improved in that: * The message shows the testing condition, not the result * The actual message has similar output than the console one
2019-08-07 22:39:01 +02:00
FAILED:
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
Why would you throw a std::string?
Improved path normalization in approvalTests.py Instead of redoing the whole line where path was found, only the directory part of the path is removed, instead of removing all of the line before the path starts. This results in slight change in how junit and sonarqube approvals come out, and significant change in how TeamCity reporter approvals come out. This latter difference is the reason for the change, as now the lines with `testFailed` and `testIgnored` messages are not completely butchered.
2022-12-31 21:17:54 +01:00
at Exception.tests.cpp:<line number>
Covered more translation possibilities
2018-02-15 19:58:49 +01:00
</error>
</testcase>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="toString on const wchar_t const pointer returns the string contents" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString on const wchar_t pointer returns the string contents" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString on wchar_t const pointer returns the string contents" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString on wchar_t returns the string contents" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString(enum class w/operator&lt;&lt;)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString(enum class)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString(enum w/operator&lt;&lt;)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="toString(enum)" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tuple&lt;>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tuple&lt;float,int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tuple&lt;int>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tuple&lt;string,string>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="tuple&lt;tuple&lt;int>,tuple&lt;>,float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="uniform samples" time="{duration}" status="run"/>
Add uniform_integer_distribution
2023-12-08 22:01:36 +01:00
<testcase classname="<exe-name>.global" name="uniform_integer_distribution can return the bounds" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Default constructed unique_ptr is empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Take ownership of allocation" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Take ownership of allocation/Plain reset deallocates" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Take ownership of allocation/Reset replaces ownership" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Release releases ownership" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Move constructor" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/Move assignment" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="unique_ptr reimplementation: basic functionality/free swap" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vec&lt;vec&lt;string,alloc>> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vector&lt;bool> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vector&lt;int,allocator> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vector&lt;int> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vector&lt;string> -> toString" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized/resizing bigger changes size and capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized/resizing smaller changes size but not capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized/resizing smaller changes size but not capacity/We can use the 'swap trick' to reset the capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized/reserving bigger changes capacity but not size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="vectors can be sized and resized/reserving smaller does not change size or capacity" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="warmup" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="weighted_average_quantile" time="{duration}" status="run"/>
Improve performance of JUnit reporter when handling passing assertions This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
2024-08-13 18:57:42 +02:00
<testcase classname="<exe-name>.global" name="xmlentitycheck" time="{duration}" status="run"/>
Add status attribute to JUnit's section reporting This brings our output inline with GTest's. We do not handle skipped tests properly, but that should be currently less important than having the attribute exist with proper value for non-skipped tests. Thanks @joda-01. Closes #1899
2020-05-15 21:00:19 +02:00
<testcase classname="<exe-name>.global" name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" time="{duration}" status="run"/>
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
<system-out>
Add tests for #1514 If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2019-03-02 21:22:41 +01:00
This would not be caught previously
Sort test ordering during Approval testing
2016-11-29 12:32:16 +01:00
A string sent directly to stdout
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
Message from section one
Message from section two
Add new SKIP macro for skipping tests at runtime (#2360) * Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2023-01-12 15:01:47 +01:00
a!
b1!
!
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</system-out>
<system-err>
Add tests for #1514 If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2019-03-02 21:22:41 +01:00
Nor would this
Changed stdout/ stderr messages to be less misleading
2013-12-19 09:07:33 +01:00
A string sent directly to stderr
Added clog test
2017-12-05 18:48:15 +01:00
A string sent to stderr via clog
Split approve file into multiple files
2013-09-27 20:01:14 +02:00
</system-err>
</testsuite>
</testsuites>
Reference in New Issue Copy Permalink