Add tests for #1514

If the regression comes back, it will only be caught by approvals, but that's better than nothing.
2024-11-08 07:09:54 +01:00 · 2019-03-02 21:22:41 +01:00 · 2019-03-02 21:22:41 +01:00 · 28db5ed4c9
commit 28db5ed4c9
parent 7d2451f119
7 changed files with 83 additions and 14 deletions
--- a/projects/SelfTest/Baselines/compact.sw.approved.txt
+++ b/projects/SelfTest/Baselines/compact.sw.approved.txt
@ -14,6 +14,9 @@ Compilation.tests.cpp:<line number>: passed: std::memcmp(uarr, "123", sizeof(uar
 Compilation.tests.cpp:<line number>: passed: std::memcmp(sarr, "456", sizeof(sarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"'
 Compilation.tests.cpp:<line number>: passed:
 Compilation.tests.cpp:<line number>: passed: h1 == h2 for: [1403 helper] == [1403 helper]
+Nor would this
+This would not be caught previously
+Tricky.tests.cpp:<line number>: failed: explicitly with 1 message: '1514'
 Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true
 Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
 Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception'
@ -1459,5 +1462,5 @@ Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
 Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
 Misc.tests.cpp:<line number>: passed:
 Misc.tests.cpp:<line number>: passed:
-Failed 69 test cases, failed 129 assertions.
+Failed 70 test cases, failed 130 assertions.

--- a/projects/SelfTest/Baselines/console.std.approved.txt
+++ b/projects/SelfTest/Baselines/console.std.approved.txt
@ -1,3 +1,5 @@
+Nor would this
+This would not be caught previously

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 <exe-name> is a <version> host application.
@ -5,6 +7,16 @@ Run with -? for options

 Randomness seeded to: 1

+-------------------------------------------------------------------------------
+#1514: stderr/stdout is not captured in tests aborted by an exception
+-------------------------------------------------------------------------------
+Tricky.tests.cpp:<line number>
+...............................................................................
+
+Tricky.tests.cpp:<line number>: FAILED:
+explicitly with message:
+  1514
+
 -------------------------------------------------------------------------------
 #748 - captures with unexpected exceptions
  outside assertions
@ -1174,6 +1186,6 @@ due to unexpected exception with message:
  Why would you throw a std::string?

 ===============================================================================
-test cases:  246 |  186 passed |  56 failed |  4 failed as expected
-assertions: 1380 | 1244 passed | 115 failed | 21 failed as expected
+test cases:  247 |  186 passed |  57 failed |  4 failed as expected
+assertions: 1381 | 1244 passed | 116 failed | 21 failed as expected

--- a/projects/SelfTest/Baselines/console.sw.approved.txt
+++ b/projects/SelfTest/Baselines/console.sw.approved.txt
@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED:
 with expansion:
  [1403 helper] == [1403 helper]

+Nor would this
+This would not be caught previously
+-------------------------------------------------------------------------------
+#1514: stderr/stdout is not captured in tests aborted by an exception
+-------------------------------------------------------------------------------
+Tricky.tests.cpp:<line number>
+...............................................................................
+
+Tricky.tests.cpp:<line number>: FAILED:
+explicitly with message:
+  1514
+
 -------------------------------------------------------------------------------
 #1548
 -------------------------------------------------------------------------------
@ -10709,6 +10721,6 @@ Misc.tests.cpp:<line number>
 Misc.tests.cpp:<line number>: PASSED:

 ===============================================================================
-test cases:  246 |  173 passed |  69 failed |  4 failed as expected
-assertions: 1394 | 1244 passed | 129 failed | 21 failed as expected
+test cases:  247 |  173 passed |  70 failed |  4 failed as expected
+assertions: 1395 | 1244 passed | 130 failed | 21 failed as expected

--- a/projects/SelfTest/Baselines/console.swa4.approved.txt
+++ b/projects/SelfTest/Baselines/console.swa4.approved.txt
@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED:
 with expansion:
  [1403 helper] == [1403 helper]

+Nor would this
+This would not be caught previously
+-------------------------------------------------------------------------------
+#1514: stderr/stdout is not captured in tests aborted by an exception
+-------------------------------------------------------------------------------
+Tricky.tests.cpp:<line number>
+...............................................................................
+
+Tricky.tests.cpp:<line number>: FAILED:
+explicitly with message:
+  1514
+
 -------------------------------------------------------------------------------
 #1548
 -------------------------------------------------------------------------------
@ -328,12 +340,7 @@ Condition.tests.cpp:<line number>: FAILED:
 with expansion:
  false

-Condition.tests.cpp:<line number>: FAILED:
-  CHECK_FALSE( true )
-with expansion:
-  !true
-
 ===============================================================================
-test cases: 16 | 13 passed | 1 failed | 2 failed as expected
+test cases: 17 | 13 passed | 2 failed | 2 failed as expected
 assertions: 40 | 33 passed | 4 failed | 3 failed as expected

--- a/projects/SelfTest/Baselines/junit.sw.approved.txt
+++ b/projects/SelfTest/Baselines/junit.sw.approved.txt
@ -4,7 +4,7 @@
    <property name="random-seed" value="1"/>
  </properties>
 loose text artifact
-  <testsuite name="<exe-name>" errors="17" failures="113" tests="1395" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
+  <testsuite name="<exe-name>" errors="17" failures="114" tests="1396" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
    <testcase classname="<exe-name>.global" name="# A test name that starts with a #" time="{duration}"/>
    <testcase classname="<exe-name>.global" name="#1005: Comparing pointer to int and long (NULL can be either on various systems)" time="{duration}"/>
    <testcase classname="<exe-name>.global" name="#1027" time="{duration}"/>
@ -13,6 +13,18 @@ loose text artifact
    <testcase classname="<exe-name>.global" name="#1238" time="{duration}"/>
    <testcase classname="<exe-name>.(Fixture_1245&lt;int, int>)" name="#1245" time="{duration}"/>
    <testcase classname="<exe-name>.global" name="#1403" time="{duration}"/>
+    <testcase classname="<exe-name>.global" name="#1514: stderr/stdout is not captured in tests aborted by an exception" time="{duration}">
+      <failure type="FAIL">
+1514
+Tricky.tests.cpp:<line number>
+      </failure>
+      <system-out>
+This would not be caught previously
+      </system-out>
+      <system-err>
+Nor would this
+      </system-err>
+    </testcase>
    <testcase classname="<exe-name>.global" name="#1548" time="{duration}"/>
    <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}">
      <error type="TEST_CASE">
@ -998,11 +1010,13 @@ Exception.tests.cpp:<line number>
    <testcase classname="<exe-name>.global" name="xmlentitycheck/embedded xml: &lt;test>it should be possible to embed xml characters, such as &lt;, &quot; or &amp;, or even whole &lt;xml>documents&lt;/xml> within an attribute&lt;/test>" time="{duration}"/>
    <testcase classname="<exe-name>.global" name="xmlentitycheck/encoded chars: these should all be encoded: &amp;&amp;&amp;&quot;&quot;&quot;&lt;&lt;&lt;&amp;&quot;&lt;&lt;&amp;&quot;" time="{duration}"/>
    <system-out>
+This would not be caught previously
 A string sent directly to stdout
 Message from section one
 Message from section two
    </system-out>
    <system-err>
+Nor would this
 A string sent directly to stderr
 A string sent to stderr via clog
    </system-err>
--- a/projects/SelfTest/Baselines/xml.sw.approved.txt
+++ b/projects/SelfTest/Baselines/xml.sw.approved.txt
@ -142,6 +142,19 @@
      </Expression>
      <OverallResult success="true"/>
    </TestCase>
+    <TestCase name="#1514: stderr/stdout is not captured in tests aborted by an exception" tags="[.][output-capture][regression]" filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" >
+      <Failure filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" >
+        1514
+      </Failure>
+      <OverallResult success="false">
+        <StdOut>
+This would not be caught previously
+        </StdOut>
+        <StdErr>
+Nor would this
+        </StdErr>
+      </OverallResult>
+    </TestCase>
    <TestCase name="#1548" tags="[compilation]" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" >
      <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" >
        <Original>
@ -12959,7 +12972,7 @@ loose text artifact
      </Section>
      <OverallResult success="true"/>
    </TestCase>
-    <OverallResults successes="1244" failures="130" expectedFailures="21"/>
+    <OverallResults successes="1244" failures="131" expectedFailures="21"/>
  </Group>
-  <OverallResults successes="1244" failures="129" expectedFailures="21"/>
+  <OverallResults successes="1244" failures="130" expectedFailures="21"/>
 </Catch>
--- a/projects/SelfTest/UsageTests/Tricky.tests.cpp
+++ b/projects/SelfTest/UsageTests/Tricky.tests.cpp
@ -19,6 +19,7 @@

 #include <stdio.h>
 #include <sstream>
+#include <iostream>

 ///////////////////////////////////////////////////////////////////////////////
 TEST_CASE
@ -426,3 +427,10 @@ TEST_CASE( "Bitfields can be captured (#1027)" ) {
    REQUIRE( y.v == 0 );
    REQUIRE( 0 == y.v );
 }
+
+TEST_CASE("#1514: stderr/stdout is not captured in tests aborted by an exception", "[output-capture][regression][.]") {
+    std::cout << "This would not be caught previously\n";
+    std::clog << "Nor would this\n";
+    // FAIL aborts the test by throwing a Catch exception
+    FAIL("1514");
+}