Add more tests for test spec parser

Originally the tests were from #1912, but as it turned out, the issue
was somewhere else. Still, the inputs provided were interesting, so
they are now part of our test suite.
This commit is contained in:
Martin Hořeňovský 2020-04-17 21:16:18 +02:00
parent bbbc7a0d7f
commit dd35430a2b
No known key found for this signature in database
GPG Key ID: DE48307B8B0D381A
11 changed files with 156 additions and 10 deletions

View File

@ -14,6 +14,7 @@ Nor would this
:test-result: FAIL #1514: stderr/stdout is not captured in tests aborted by an exception :test-result: FAIL #1514: stderr/stdout is not captured in tests aborted by an exception
:test-result: PASS #1548 :test-result: PASS #1548
:test-result: PASS #1905 -- test spec parser properly clears internal state between compound tests :test-result: PASS #1905 -- test spec parser properly clears internal state between compound tests
:test-result: PASS #1912 -- test spec parser handles escaping
:test-result: XFAIL #748 - captures with unexpected exceptions :test-result: XFAIL #748 - captures with unexpected exceptions
:test-result: PASS #809 :test-result: PASS #809
:test-result: PASS #833 :test-result: PASS #833

View File

@ -27,6 +27,10 @@ Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeLis
CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec . char")) for: true CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec . char")) for: true
CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec , char")) for: true CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec , char")) for: true
CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false
CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec {a} char)")) for: true
CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec [a] char)")) for: true
CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase("differs but has similar tag", "[a]"))) for: !false
CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec \ char)")) for: true
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception'
Exception.tests.cpp:<line number>: passed: thisThrows() with 1 message: 'answer := 42' Exception.tests.cpp:<line number>: passed: thisThrows() with 1 message: 'answer := 42'

View File

@ -1380,6 +1380,6 @@ due to unexpected exception with message:
Why would you throw a std::string? Why would you throw a std::string?
=============================================================================== ===============================================================================
test cases: 333 | 259 passed | 70 failed | 4 failed as expected test cases: 334 | 260 passed | 70 failed | 4 failed as expected
assertions: 1876 | 1724 passed | 131 failed | 21 failed as expected assertions: 1880 | 1728 passed | 131 failed | 21 failed as expected

View File

@ -218,6 +218,40 @@ CmdLine.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
!false !false
-------------------------------------------------------------------------------
#1912 -- test spec parser handles escaping
Various parentheses
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec {a} char)")) )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec [a] char)")) )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_FALSE( spec.matches(*fakeTestCase("differs but has similar tag", "[a]")) )
with expansion:
!false
-------------------------------------------------------------------------------
#1912 -- test spec parser handles escaping
backslash in test name
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec \ char)")) )
with expansion:
true
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
#748 - captures with unexpected exceptions #748 - captures with unexpected exceptions
outside assertions outside assertions
@ -14679,6 +14713,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED: Misc.tests.cpp:<line number>: PASSED:
=============================================================================== ===============================================================================
test cases: 333 | 243 passed | 86 failed | 4 failed as expected test cases: 334 | 244 passed | 86 failed | 4 failed as expected
assertions: 1893 | 1724 passed | 148 failed | 21 failed as expected assertions: 1897 | 1728 passed | 148 failed | 21 failed as expected

View File

@ -218,6 +218,40 @@ CmdLine.tests.cpp:<line number>: PASSED:
with expansion: with expansion:
!false !false
-------------------------------------------------------------------------------
#1912 -- test spec parser handles escaping
Various parentheses
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec {a} char)")) )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec [a] char)")) )
with expansion:
true
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_FALSE( spec.matches(*fakeTestCase("differs but has similar tag", "[a]")) )
with expansion:
!false
-------------------------------------------------------------------------------
#1912 -- test spec parser handles escaping
backslash in test name
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................
CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE( spec.matches(*fakeTestCase(R"(spec \ char)")) )
with expansion:
true
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
#748 - captures with unexpected exceptions #748 - captures with unexpected exceptions
outside assertions outside assertions
@ -398,6 +432,6 @@ Condition.tests.cpp:<line number>: FAILED:
CHECK( true != true ) CHECK( true != true )
=============================================================================== ===============================================================================
test cases: 21 | 16 passed | 3 failed | 2 failed as expected test cases: 22 | 17 passed | 3 failed | 2 failed as expected
assertions: 46 | 39 passed | 4 failed | 3 failed as expected assertions: 50 | 43 passed | 4 failed | 3 failed as expected

View File

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact <testsuitesloose text artifact
> >
<testsuite name="<exe-name>" errors="17" failures="132" tests="1894" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testsuite name="<exe-name>" errors="17" failures="132" tests="1898" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties> <properties>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/> <property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
<property name="random-seed" value="1"/> <property name="random-seed" value="1"/>
@ -32,6 +32,8 @@ Nor would this
</testcase> </testcase>
<testcase classname="<exe-name>.global" name="#1548" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1548" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1905 -- test spec parser properly clears internal state between compound tests" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1905 -- test spec parser properly clears internal state between compound tests" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/Various parentheses" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/backslash in test name" time="{duration}"/>
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}"> <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}">
<error type="TEST_CASE"> <error type="TEST_CASE">
FAILED: FAILED:

View File

@ -3,6 +3,8 @@
> >
<file path="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp"> <file path="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp">
<testCase name="#1905 -- test spec parser properly clears internal state between compound tests" duration="{duration}"/> <testCase name="#1905 -- test spec parser properly clears internal state between compound tests" duration="{duration}"/>
<testCase name="#1912 -- test spec parser handles escaping/Various parentheses" duration="{duration}"/>
<testCase name="#1912 -- test spec parser handles escaping/backslash in test name" duration="{duration}"/>
<testCase name="Parse test names and tags/Empty test spec should have no filters" duration="{duration}"/> <testCase name="Parse test names and tags/Empty test spec should have no filters" duration="{duration}"/>
<testCase name="Parse test names and tags/Test spec from empty string should have no filters" duration="{duration}"/> <testCase name="Parse test names and tags/Test spec from empty string should have no filters" duration="{duration}"/>
<testCase name="Parse test names and tags/Test spec from just a comma should have no filters" duration="{duration}"/> <testCase name="Parse test names and tags/Test spec from just a comma should have no filters" duration="{duration}"/>

View File

@ -52,6 +52,14 @@ ok {test-number} - spec.matches(*fakeTestCase("spec . char")) for: true
ok {test-number} - spec.matches(*fakeTestCase("spec , char")) for: true ok {test-number} - spec.matches(*fakeTestCase("spec , char")) for: true
# #1905 -- test spec parser properly clears internal state between compound tests # #1905 -- test spec parser properly clears internal state between compound tests
ok {test-number} - !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false ok {test-number} - !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false
# #1912 -- test spec parser handles escaping
ok {test-number} - spec.matches(*fakeTestCase(R"(spec {a} char)")) for: true
# #1912 -- test spec parser handles escaping
ok {test-number} - spec.matches(*fakeTestCase(R"(spec [a] char)")) for: true
# #1912 -- test spec parser handles escaping
ok {test-number} - !(spec.matches(*fakeTestCase("differs but has similar tag", "[a]"))) for: !false
# #1912 -- test spec parser handles escaping
ok {test-number} - spec.matches(*fakeTestCase(R"(spec \ char)")) for: true
# #748 - captures with unexpected exceptions # #748 - captures with unexpected exceptions
not ok {test-number} - unexpected exception with message: 'answer := 42' with 1 message: 'expected exception' not ok {test-number} - unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
# #748 - captures with unexpected exceptions # #748 - captures with unexpected exceptions
@ -3778,5 +3786,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} - ok {test-number} -
# xmlentitycheck # xmlentitycheck
ok {test-number} - ok {test-number} -
1..1885 1..1889

View File

@ -30,6 +30,8 @@ Tricky.tests.cpp:<line number>|nexplicit failure with message:|n "1514"']
##teamcity[testFinished name='#1548' duration="{duration}"] ##teamcity[testFinished name='#1548' duration="{duration}"]
##teamcity[testStarted name='#1905 -- test spec parser properly clears internal state between compound tests'] ##teamcity[testStarted name='#1905 -- test spec parser properly clears internal state between compound tests']
##teamcity[testFinished name='#1905 -- test spec parser properly clears internal state between compound tests' duration="{duration}"] ##teamcity[testFinished name='#1905 -- test spec parser properly clears internal state between compound tests' duration="{duration}"]
##teamcity[testStarted name='#1912 -- test spec parser handles escaping']
##teamcity[testFinished name='#1912 -- test spec parser handles escaping' duration="{duration}"]
##teamcity[testStarted name='#748 - captures with unexpected exceptions'] ##teamcity[testStarted name='#748 - captures with unexpected exceptions']
Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"- failure ignore as test marked as |'ok to fail|'|n'] Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"- failure ignore as test marked as |'ok to fail|'|n']
Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"|n REQUIRE_NOTHROW( thisThrows() )|nwith expansion:|n thisThrows()|n- failure ignore as test marked as |'ok to fail|'|n'] Exception.tests.cpp:<line number>|nunexpected exception with messages:|n "answer := 42"|n "expected exception"|n REQUIRE_NOTHROW( thisThrows() )|nwith expansion:|n thisThrows()|n- failure ignore as test marked as |'ok to fail|'|n']

View File

@ -229,6 +229,47 @@ Nor would this
</Expression> </Expression>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<TestCase name="#1912 -- test spec parser handles escaping" tags="[command-line][test-spec]" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Section name="Various parentheses" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
spec.matches(*fakeTestCase(R"(spec {a} char)"))
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
spec.matches(*fakeTestCase(R"(spec [a] char)"))
</Original>
<Expanded>
true
</Expanded>
</Expression>
<Expression success="true" type="REQUIRE_FALSE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
!(spec.matches(*fakeTestCase("differs but has similar tag", "[a]")))
</Original>
<Expanded>
!false
</Expanded>
</Expression>
<OverallResults successes="3" failures="0" expectedFailures="0"/>
</Section>
<Section name="backslash in test name" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
<Original>
spec.matches(*fakeTestCase(R"(spec \ char)"))
</Original>
<Expanded>
true
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0"/>
</Section>
<OverallResult success="true"/>
</TestCase>
<TestCase name="#748 - captures with unexpected exceptions" tags="[!shouldfail][!throws][.][failing]" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" > <TestCase name="#748 - captures with unexpected exceptions" tags="[!shouldfail][!throws][.][failing]" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" >
<Section name="outside assertions" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" > <Section name="outside assertions" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" >
<Info> <Info>
@ -17595,7 +17636,7 @@ loose text artifact
</Section> </Section>
<OverallResult success="true"/> <OverallResult success="true"/>
</TestCase> </TestCase>
<OverallResults successes="1724" failures="149" expectedFailures="21"/> <OverallResults successes="1728" failures="149" expectedFailures="21"/>
</Group> </Group>
<OverallResults successes="1724" failures="148" expectedFailures="21"/> <OverallResults successes="1728" failures="148" expectedFailures="21"/>
</Catch> </Catch>

View File

@ -307,6 +307,24 @@ TEST_CASE("#1905 -- test spec parser properly clears internal state between comp
REQUIRE_FALSE(spec.matches(*fakeTestCase(R"(spec \, char)"))); REQUIRE_FALSE(spec.matches(*fakeTestCase(R"(spec \, char)")));
} }
TEST_CASE("#1912 -- test spec parser handles escaping", "[command-line][test-spec]") {
using Catch::parseTestSpec;
using Catch::TestSpec;
SECTION("Various parentheses") {
TestSpec spec = parseTestSpec(R"(spec {a} char,spec \[a] char)");
REQUIRE(spec.matches(*fakeTestCase(R"(spec {a} char)")));
REQUIRE(spec.matches(*fakeTestCase(R"(spec [a] char)")));
REQUIRE_FALSE(spec.matches(*fakeTestCase("differs but has similar tag", "[a]")));
}
SECTION("backslash in test name") {
TestSpec spec = parseTestSpec(R"(spec \\ char)");
REQUIRE(spec.matches(*fakeTestCase(R"(spec \ char)")));
}
}
TEST_CASE( "Process can be configured on command line", "[config][command-line]" ) { TEST_CASE( "Process can be configured on command line", "[config][command-line]" ) {
using namespace Catch::Matchers; using namespace Catch::Matchers;