Add tests for #1514

If the regression comes back, it will only be caught by approvals,
but that's better than nothing.
This commit is contained in:
Martin Hořeňovský 2019-03-02 21:22:41 +01:00
parent 7d2451f119
commit 28db5ed4c9
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
7 changed files with 83 additions and 14 deletions

View File

@ -14,6 +14,9 @@ Compilation.tests.cpp:<line number>: passed: std::memcmp(uarr, "123", sizeof(uar
Compilation.tests.cpp:<line number>: passed: std::memcmp(sarr, "456", sizeof(sarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"' Compilation.tests.cpp:<line number>: passed: std::memcmp(sarr, "456", sizeof(sarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"'
Compilation.tests.cpp:<line number>: passed: Compilation.tests.cpp:<line number>: passed:
Compilation.tests.cpp:<line number>: passed: h1 == h2 for: [1403 helper] == [1403 helper] Compilation.tests.cpp:<line number>: passed: h1 == h2 for: [1403 helper] == [1403 helper]
Nor would this
This would not be caught previously
Tricky.tests.cpp:<line number>: failed: explicitly with 1 message: '1514'
Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception'
@ -1459,5 +1462,5 @@ Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5 Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed:
Failed 69 test cases, failed 129 assertions. Failed 70 test cases, failed 130 assertions.

View File

@ -1,3 +1,5 @@
Nor would this
This would not be caught previously
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<exe-name> is a <version> host application. <exe-name> is a <version> host application.
@ -5,6 +7,16 @@ Run with -? for options
Randomness seeded to: 1 Randomness seeded to: 1
-------------------------------------------------------------------------------
#1514: stderr/stdout is not captured in tests aborted by an exception
-------------------------------------------------------------------------------
Tricky.tests.cpp:<line number>
...............................................................................
Tricky.tests.cpp:<line number>: FAILED:
explicitly with message:
1514
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
#748 - captures with unexpected exceptions #748 - captures with unexpected exceptions
outside assertions outside assertions
@ -1174,6 +1186,6 @@ due to unexpected exception with message:
Why would you throw a std::string? Why would you throw a std::string?
=============================================================================== ===============================================================================
test cases: 246 | 186 passed | 56 failed | 4 failed as expected test cases: 247 | 186 passed | 57 failed | 4 failed as expected
assertions: 1380 | 1244 passed | 115 failed | 21 failed as expected assertions: 1381 | 1244 passed | 116 failed | 21 failed as expected

View File

@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
[1403 helper] == [1403 helper] [1403 helper] == [1403 helper]
Nor would this
This would not be caught previously
-------------------------------------------------------------------------------
#1514: stderr/stdout is not captured in tests aborted by an exception
-------------------------------------------------------------------------------
Tricky.tests.cpp:<line number>
...............................................................................
Tricky.tests.cpp:<line number>: FAILED:
explicitly with message:
1514
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
#1548 #1548
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -10709,6 +10721,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 246 | 173 passed | 69 failed | 4 failed as expected test cases: 247 | 173 passed | 70 failed | 4 failed as expected
assertions: 1394 | 1244 passed | 129 failed | 21 failed as expected assertions: 1395 | 1244 passed | 130 failed | 21 failed as expected

View File

@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
[1403 helper] == [1403 helper] [1403 helper] == [1403 helper]
Nor would this
This would not be caught previously
-------------------------------------------------------------------------------
#1514: stderr/stdout is not captured in tests aborted by an exception
-------------------------------------------------------------------------------
Tricky.tests.cpp:<line number>
...............................................................................
Tricky.tests.cpp:<line number>: FAILED:
explicitly with message:
1514
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
#1548 #1548
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
@ -328,12 +340,7 @@ Condition.tests.cpp:<line number>: FAILED:
with expansion: with expansion:
false false
Condition.tests.cpp:<line number>: FAILED:
CHECK_FALSE( true )
with expansion:
!true
=============================================================================== ===============================================================================
test cases: 16 | 13 passed | 1 failed | 2 failed as expected test cases: 17 | 13 passed | 2 failed | 2 failed as expected
assertions: 40 | 33 passed | 4 failed | 3 failed as expected assertions: 40 | 33 passed | 4 failed | 3 failed as expected

View File

@ -4,7 +4,7 @@
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
</properties> </properties>
loose text artifact loose text artifact
<testsuite name="<exe-name>" errors="17" failures="113" tests="1395" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="114" tests="1396" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testcase classname="<exe-name>.global" name="# A test name that starts with a #" time="{duration}"/> <testcase classname="<exe-name>.global" name="# A test name that starts with a #" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1005: Comparing pointer to int and long (NULL can be either on various systems)" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1005: Comparing pointer to int and long (NULL can be either on various systems)" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1027" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1027" time="{duration}"/>
@ -13,6 +13,18 @@ loose text artifact
<testcase classname="<exe-name>.global" name="#1238" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1238" time="{duration}"/>
<testcase classname="<exe-name>.(Fixture_1245&lt;int, int>)" name="#1245" time="{duration}"/> <testcase classname="<exe-name>.(Fixture_1245&lt;int, int>)" name="#1245" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1403" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1403" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1514: stderr/stdout is not captured in tests aborted by an exception" time="{duration}">
<failure type="FAIL">
1514
Tricky.tests.cpp:<line number>
</failure>
<system-out>
This would not be caught previously
</system-out>
<system-err>
Nor would this
</system-err>
</testcase>
<testcase classname="<exe-name>.global" name="#1548" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1548" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}"> <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}">
<error type="TEST_CASE"> <error type="TEST_CASE">
@ -998,11 +1010,13 @@ Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" time="{duration}"/> <testcase classname="<exe-name>.global" name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" time="{duration}"/>
<testcase classname="<exe-name>.global" name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" time="{duration}"/> <testcase classname="<exe-name>.global" name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" time="{duration}"/>
<system-out> <system-out>
This would not be caught previously
A string sent directly to stdout A string sent directly to stdout
Message from section one Message from section one
Message from section two Message from section two
</system-out> </system-out>
<system-err> <system-err>
Nor would this
A string sent directly to stderr A string sent directly to stderr
A string sent to stderr via clog A string sent to stderr via clog
</system-err> </system-err>

View File

@ -142,6 +142,19 @@
</Expression> </Expression>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<TestCase name="#1514: stderr/stdout is not captured in tests aborted by an exception" tags="[.][output-capture][regression]" filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" >
<Failure filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" >
1514
</Failure>
<OverallResult success="false">
<StdOut>
This would not be caught previously
</StdOut>
<StdErr>
Nor would this
</StdErr>
</OverallResult>
</TestCase>
<TestCase name="#1548" tags="[compilation]" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" > <TestCase name="#1548" tags="[compilation]" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" > <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Original> <Original>
@ -12959,7 +12972,7 @@ loose text artifact
</Section> </Section>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<OverallResults successes="1244" failures="130" expectedFailures="21"/> <OverallResults successes="1244" failures="131" expectedFailures="21"/>
</Group> </Group>
<OverallResults successes="1244" failures="129" expectedFailures="21"/> <OverallResults successes="1244" failures="130" expectedFailures="21"/>
</Catch> </Catch>

View File

@ -19,6 +19,7 @@
#include <stdio.h> #include <stdio.h>
#include <sstream> #include <sstream>
#include <iostream>
/////////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////////
TEST_CASE TEST_CASE
@ -426,3 +427,10 @@ TEST_CASE( "Bitfields can be captured (#1027)" ) {
REQUIRE( y.v == 0 ); REQUIRE( y.v == 0 );
REQUIRE( 0 == y.v ); REQUIRE( 0 == y.v );
} }
TEST_CASE("#1514: stderr/stdout is not captured in tests aborted by an exception", "[output-capture][regression][.]") {
std::cout << "This would not be caught previously\n";
std::clog << "Nor would this\n";
// FAIL aborts the test by throwing a Catch exception
FAIL("1514");
}