3rd-party/catch2
1
0
Fork 0
mirror of https://github.com/catchorg/Catch2.git synced 2024-11-23 05:46:11 +01:00
Code Issues Packages Projects Releases Wiki Activity
33ffc3b6fc
catch2/tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp

719 lines
26 KiB
C++
Raw Normal View History

Use the new licence header in SelfTest's cpp files
2022-01-29 00:03:43 +01:00
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE_1_0.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
#include <catch2/catch_config.hpp>
Standardize header names and file locations This is both a really big and a really small commit. It is small in that it only contains renaming, moving and modification of include directives caused by this. It is really big in the obvious way of touching something like 200 files. The new rules for naming files is simple: headers use the `.hpp` extension. The rules for physical file layout is still kinda in progress, but the basics are also simple: * Significant parts of functionality get their own subfolder * Benchmarking is in `catch2/benchmark` * Matchers are in `catch2/matchers` * Generators are in `catch2/generators` * Reporters are in `catch2/reporters` * Baseline testing facilities are in `catch2/` * Various top level folders also contain `internal` subfolder, with files that users probably do not want to include directly, at least not until they have to write something like their own reporter. * The exact files in these subfolders is likely to change later on Note that while some includes were cleaned up in this commit, it is only the low hanging fruit and further cleanup using automatic tooling will happen later. Also note that various include guards, copyright notices and file headers will also be standardized later, rather than in this commit.
2020-03-30 10:34:21 +02:00
#include <catch2/catch_approx.hpp>
Start using piecemeal includes in test files
2020-01-20 23:24:04 +01:00
#include <catch2/catch_test_macros.hpp>
Standardize matcher headers to use .hpp suffix
2020-03-01 19:59:18 +01:00
#include <catch2/matchers/catch_matchers_string.hpp>
Standardize header names and file locations This is both a really big and a really small commit. It is small in that it only contains renaming, moving and modification of include directives caused by this. It is really big in the obvious way of touching something like 200 files. The new rules for naming files is simple: headers use the `.hpp` extension. The rules for physical file layout is still kinda in progress, but the basics are also simple: * Significant parts of functionality get their own subfolder * Benchmarking is in `catch2/benchmark` * Matchers are in `catch2/matchers` * Generators are in `catch2/generators` * Reporters are in `catch2/reporters` * Baseline testing facilities are in `catch2/` * Various top level folders also contain `internal` subfolder, with files that users probably do not want to include directly, at least not until they have to write something like their own reporter. * The exact files in these subfolders is likely to change later on Note that while some includes were cleaned up in this commit, it is only the low hanging fruit and further cleanup using automatic tooling will happen later. Also note that various include guards, copyright notices and file headers will also be standardized later, rather than in this commit.
2020-03-30 10:34:21 +02:00
#include <catch2/internal/catch_test_spec_parser.hpp>
Push down handling of default reporter to Config's constructor This simplifies the handling of default reporter in console parsing, at the cost of making `Config`'s constructor responsible for more things.
2022-02-17 20:44:27 +01:00
#include <catch2/catch_user_config.hpp>
Standardize header names and file locations This is both a really big and a really small commit. It is small in that it only contains renaming, moving and modification of include directives caused by this. It is really big in the obvious way of touching something like 200 files. The new rules for naming files is simple: headers use the `.hpp` extension. The rules for physical file layout is still kinda in progress, but the basics are also simple: * Significant parts of functionality get their own subfolder * Benchmarking is in `catch2/benchmark` * Matchers are in `catch2/matchers` * Generators are in `catch2/generators` * Reporters are in `catch2/reporters` * Baseline testing facilities are in `catch2/` * Various top level folders also contain `internal` subfolder, with files that users probably do not want to include directly, at least not until they have to write something like their own reporter. * The exact files in these subfolders is likely to change later on Note that while some includes were cleaned up in this commit, it is only the low hanging fruit and further cleanup using automatic tooling will happen later. Also note that various include guards, copyright notices and file headers will also be standardized later, rather than in this commit.
2020-03-30 10:34:21 +02:00
#include <catch2/catch_test_case_info.hpp>
#include <catch2/internal/catch_commandline.hpp>
#include <catch2/generators/catch_generators.hpp>
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Fixed an issue on some compilers with implicit conversion from nullptr to Ptr - also cleaned up some warnings to do with CATCH_NULL
2015-07-03 19:27:36 +02:00
#ifdef __clang__
# pragma clang diagnostic ignored "-Wc++98-compat"
#endif
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
namespace {
auto fakeTestCase(const char* name, const char* desc = "") { return Catch::makeTestCaseInfo("", { name, desc }, CATCH_INTERNAL_LINEINFO); }
}
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Split [.foo] into [.][foo] when parsing test specs b77cec05c0 fixed this problem for tagging tests, so that a test case tagged with `[.foo]` would be parsed as tagged with `[.][foo]`. This does the same for the test spec parsing. Fixes #1798
2019-11-05 23:28:47 +01:00
TEST_CASE( "Parse test names and tags", "[command-line][test-spec]" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
using Catch::parseTestSpec;
using Catch::TestSpec;
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
auto tcA = fakeTestCase( "a" );
auto tcB = fakeTestCase( "b", "[one][x]" );
auto tcC = fakeTestCase( "longer name with spaces", "[two][three][.][x]" );
auto tcD = fakeTestCase( "zlonger name with spacesz" );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Empty test spec should have no filters" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec;
CHECK( spec.hasFilters() == false );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Test spec from empty string should have no filters" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "" );
CHECK( spec.hasFilters() == false );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Test spec from just a comma should have no filters" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "," );
CHECK( spec.hasFilters() == false );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Test spec from name should have one filter" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "b" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Test spec from quoted name should have one filter" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "\"b\"" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Test spec from name should have one filter" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "b" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Wildcard at the start" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "*spaces" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
CHECK( parseTestSpec( "*a" ).matches( *tcA ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Wildcard at the end" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "long*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
CHECK( parseTestSpec( "a*" ).matches( *tcA ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Wildcard at both ends" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "*name*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == true );
CHECK( parseTestSpec( "*a*" ).matches( *tcA ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Redundant wildcard at the start" ) {
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
TestSpec spec = parseTestSpec( "*a" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == false );
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Redundant wildcard at the end" ) {
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
TestSpec spec = parseTestSpec( "a*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == false );
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Redundant wildcard at both ends" ) {
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
TestSpec spec = parseTestSpec( "*a*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == false );
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Wildcard at both ends, redundant at start" ) {
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
TestSpec spec = parseTestSpec( "*longer*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == true );
Fixed issue with wildcards at the start of a string
2014-05-19 19:07:53 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Just wildcard" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Single tag" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "[one]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Single tag, two matches" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "[x]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Two tags" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "[two][x]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Two tags, spare separated" ) {
Fixed space separated lists of test specs - they form an AND expression. They were forming an OR expression due to changes made to fix -f - so that had to be fixed differently
2014-05-20 19:28:19 +02:00
TestSpec spec = parseTestSpec( "[two] [x]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
Fixed space separated lists of test specs - they form an AND expression. They were forming an OR expression due to changes made to fix -f - so that had to be fixed differently
2014-05-20 19:28:19 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Wildcarded name and tag" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "*name*[x]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "Single tag exclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~[one]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "One tag exclusion and one tag inclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~[two][x]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "One tag exclusion and one wldcarded name inclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~[two]*name*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "One tag exclusion, using exclude:, and one wldcarded name inclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "exclude:[two]*name*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "name exclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~b" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == true );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "wildcarded name exclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~*name*" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "wildcarded name exclusion with tag inclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "~*name*,[three]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "wildcarded name exclusion, using exclude:, with tag inclusion" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "exclude:*name*,[three]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == true );
CHECK( spec.matches( *tcB ) == true );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "two wildcarded names" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "\"longer*\"\"*spaces\"" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == true );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "empty tag" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "[]" );
CHECK( spec.hasFilters() == false );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "empty quoted name" ) {
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
TestSpec spec = parseTestSpec( "\"\"" );
CHECK( spec.hasFilters() == false );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == false );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
Removed all (that I could find) redundant second macro args to TEST_CASE and SECTION - now we can rely on variadic macros
2017-07-13 10:20:37 +02:00
SECTION( "quoted string followed by tag exclusion" ) {
Switched over to new name/ tag parser
2014-05-16 19:24:07 +02:00
TestSpec spec = parseTestSpec( "\"*name*\"~[.]" );
CHECK( spec.hasFilters() == true );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *tcA ) == false );
CHECK( spec.matches( *tcB ) == false );
CHECK( spec.matches( *tcC ) == false );
CHECK( spec.matches( *tcD ) == true );
Switched over to new name/ tag parser
2014-05-16 19:24:07 +02:00
}
Ignore leading/trailing whitespace in test/section specs The leading/trailing whitespace is problematic because of e.g. `WHEN` macro having preceeding whitespace for alignment, and it is generally messy. Credits to Phil who did lot of the original work. Closes #1708
2019-09-09 11:30:45 +02:00
SECTION( "Leading and trailing spaces in test spec" ) {
TestSpec spec = parseTestSpec( "\" aardvark \"" );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *fakeTestCase( " aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( " aardvark" ) ) );
CHECK( spec.matches( *fakeTestCase( " aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( "aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( "aardvark" ) ) );
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
Ignore leading/trailing whitespace in test/section specs The leading/trailing whitespace is problematic because of e.g. `WHEN` macro having preceeding whitespace for alignment, and it is generally messy. Credits to Phil who did lot of the original work. Closes #1708
2019-09-09 11:30:45 +02:00
}
SECTION( "Leading and trailing spaces in test name" ) {
TestSpec spec = parseTestSpec( "aardvark" );
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
CHECK( spec.matches( *fakeTestCase( " aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( " aardvark" ) ) );
CHECK( spec.matches( *fakeTestCase( " aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( "aardvark " ) ) );
CHECK( spec.matches( *fakeTestCase( "aardvark" ) ) );
Split [.foo] into [.][foo] when parsing test specs b77cec05c0 fixed this problem for tagging tests, so that a test case tagged with `[.foo]` would be parsed as tagged with `[.][foo]`. This does the same for the test spec parsing. Fixes #1798
2019-11-05 23:28:47 +01:00
}
SECTION("Shortened hide tags are split apart when parsing") {
TestSpec spec = parseTestSpec("[.foo]");
CHECK(spec.matches(*fakeTestCase("hidden and foo", "[.][foo]")));
CHECK_FALSE(spec.matches(*fakeTestCase("only foo", "[foo]")));
}
SECTION("Shortened hide tags also properly handle exclusion") {
TestSpec spec = parseTestSpec("~[.foo]");
CHECK_FALSE(spec.matches(*fakeTestCase("hidden and foo", "[.][foo]")));
CHECK_FALSE(spec.matches(*fakeTestCase("only foo", "[foo]")));
CHECK_FALSE(spec.matches(*fakeTestCase("only hidden", "[.]")));
CHECK(spec.matches(*fakeTestCase("neither foo nor hidden", "[bar]")));
Ignore leading/trailing whitespace in test/section specs The leading/trailing whitespace is problematic because of e.g. `WHEN` macro having preceeding whitespace for alignment, and it is generally messy. Credits to Phil who did lot of the original work. Closes #1708
2019-09-09 11:30:45 +02:00
}
First cut of new test spec parser code
2014-05-16 07:50:00 +02:00
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
Fix bug in test spec parser handling of escaping in ORed patterns It did not clear out all of its internal state when switching from one pattern to another, so when it should've escaped `,`, it took its position from its position in the original user-provided string, rather than its position in the current pattern. Fixes #1905
2020-04-12 18:44:42 +02:00
TEST_CASE("#1905 -- test spec parser properly clears internal state between compound tests", "[command-line][test-spec]") {
using Catch::parseTestSpec;
using Catch::TestSpec;
// We ask for one of 2 different tests and the latter one of them has a , in name that needs escaping
TestSpec spec = parseTestSpec(R"("spec . char","spec \, char")");
REQUIRE(spec.matches(*fakeTestCase("spec . char")));
REQUIRE(spec.matches(*fakeTestCase("spec , char")));
REQUIRE_FALSE(spec.matches(*fakeTestCase(R"(spec \, char)")));
}
Add more tests for test spec parser Originally the tests were from #1912, but as it turned out, the issue was somewhere else. Still, the inputs provided were interesting, so they are now part of our test suite.
2020-04-17 21:16:18 +02:00
TEST_CASE("#1912 -- test spec parser handles escaping", "[command-line][test-spec]") {
using Catch::parseTestSpec;
using Catch::TestSpec;
SECTION("Various parentheses") {
TestSpec spec = parseTestSpec(R"(spec {a} char,spec \[a] char)");
REQUIRE(spec.matches(*fakeTestCase(R"(spec {a} char)")));
REQUIRE(spec.matches(*fakeTestCase(R"(spec [a] char)")));
REQUIRE_FALSE(spec.matches(*fakeTestCase("differs but has similar tag", "[a]")));
}
SECTION("backslash in test name") {
TestSpec spec = parseTestSpec(R"(spec \\ char)");
REQUIRE(spec.matches(*fakeTestCase(R"(spec \ char)")));
}
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
TEST_CASE( "Process can be configured on command line", "[config][command-line]" ) {
using namespace Catch::Matchers;
Catch::ConfigData config;
auto cli = Catch::makeCommandLineParser(config);
SECTION("empty args don't cause a crash") {
auto result = cli.parse({""});
CHECK(result);
CHECK(config.processName == "");
}
SECTION("default - no arguments") {
auto result = cli.parse({"test"});
CHECK(result);
CHECK(config.processName == "test");
CHECK(config.shouldDebugBreak == false);
CHECK(config.abortAfter == -1);
CHECK(config.noThrow == false);
Push down handling of default reporter to Config's constructor This simplifies the handling of default reporter in console parsing, at the cost of making `Config`'s constructor responsible for more things.
2022-02-17 20:44:27 +01:00
CHECK( config.reporterSpecifications.empty() );
https://github.com/catchorg/Catch2/issues/1175 - don't list hidden tests by default
2018-02-08 23:18:32 +01:00
Catch::Config cfg(config);
CHECK_FALSE(cfg.hasTestFilters());
Push down handling of default reporter to Config's constructor This simplifies the handling of default reporter in console parsing, at the cost of making `Config`'s constructor responsible for more things.
2022-02-17 20:44:27 +01:00
// The Config is responsible for mixing in the default reporter
auto expectedReporter =
#if defined( CATCH_CONFIG_DEFAULT_REPORTER )
CATCH_CONFIG_DEFAULT_REPORTER
#else
"console"
#endif
;
CHECK( cfg.getReportersAndOutputFiles().size() == 1 );
CHECK( cfg.getReportersAndOutputFiles()[0] ==
Catch::ConfigData::ReporterAndFile{ expectedReporter, {} } );
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("test lists") {
Deprecated description in SECTION (still accepts it, for now, but doesn't use it anywhere)
2018-06-25 20:04:29 +02:00
SECTION("Specify one test case using") {
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
auto result = cli.parse({"test", "test1"});
CHECK(result);
Catch::Config cfg(config);
https://github.com/catchorg/Catch2/issues/1175 - don't list hidden tests by default
2018-02-08 23:18:32 +01:00
REQUIRE(cfg.hasTestFilters());
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
REQUIRE(cfg.testSpec().matches(*fakeTestCase("notIncluded")) == false);
REQUIRE(cfg.testSpec().matches(*fakeTestCase("test1")));
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("Specify one test case exclusion using exclude:") {
auto result = cli.parse({"test", "exclude:test1"});
CHECK(result);
Catch::Config cfg(config);
https://github.com/catchorg/Catch2/issues/1175 - don't list hidden tests by default
2018-02-08 23:18:32 +01:00
REQUIRE(cfg.hasTestFilters());
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
REQUIRE(cfg.testSpec().matches(*fakeTestCase("test1")) == false);
REQUIRE(cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")));
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("Specify one test case exclusion using ~") {
auto result = cli.parse({"test", "~test1"});
CHECK(result);
Catch::Config cfg(config);
https://github.com/catchorg/Catch2/issues/1175 - don't list hidden tests by default
2018-02-08 23:18:32 +01:00
REQUIRE(cfg.hasTestFilters());
Do not copy around TestCaseInfo Now a `TEST_CASE` macro should create a single TestCaseInfo and then it should never be copied around. This, together with latter changes, should significantly decrease the number of allocations made before `main` is even entered.
2019-11-04 21:35:57 +01:00
REQUIRE(cfg.testSpec().matches(*fakeTestCase("test1")) == false);
REQUIRE(cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")));
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
}
SECTION("reporter") {
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
using vec_ReporterAndFile = std::vector<Catch::ConfigData::ReporterAndFile>;
using namespace std::string_literals;
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
SECTION("-r/console") {
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
auto result = cli.parse({"test", "-r", "console"});
CAPTURE(result.errorMessage());
CHECK(result);
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} });
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("-r/xml") {
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
auto result = cli.parse({"test", "-r", "xml"});
CAPTURE(result.errorMessage());
CHECK(result);
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} });
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("--reporter/junit") {
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
auto result = cli.parse({"test", "--reporter", "junit"});
CAPTURE(result.errorMessage());
CHECK(result);
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} });
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
Catch's CLI now checks whether requested reporter exists Fixes #1351
2018-10-22 15:59:01 +02:00
SECTION("must match one of the available ones") {
auto result = cli.parse({"test", "--reporter", "unsupported"});
CHECK(!result);
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Unrecognized reporter"));
Catch's CLI now checks whether requested reporter exists Fixes #1351
2018-10-22 15:59:01 +02:00
}
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
SECTION("With output file") {
auto result = cli.parse({ "test", "-r", "console::out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} });
}
SECTION("With Windows-like absolute path as output file") {
auto result = cli.parse({ "test", "-r", "console::C:\\Temp\\out.txt" });
CAPTURE(result.errorMessage());
CHECK(result);
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} });
}
SECTION("Output file cannot be empty") {
auto result = cli.parse({"test", "--reporter", "console::"});
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("empty filename"));
}
SECTION("Multiple reporters") {
SECTION("All with output files") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} });
}
SECTION("Mixed output files and default output") {
CHECK(cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }));
REQUIRE(config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} });
}
SECTION("cannot have multiple reporters with default output") {
auto result = cli.parse({ "test", "-r", "console", "-r", "xml::output.xml", "-r", "junit" });
CHECK(!result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file."));
}
}
Catch's CLI now checks whether requested reporter exists Fixes #1351
2018-10-22 15:59:01 +02:00
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
SECTION("debugger") {
SECTION("-b") {
CHECK(cli.parse({"test", "-b"}));
REQUIRE(config.shouldDebugBreak == true);
}
SECTION("--break") {
CHECK(cli.parse({"test", "--break"}));
REQUIRE(config.shouldDebugBreak);
}
}
SECTION("abort") {
SECTION("-a aborts after first failure") {
CHECK(cli.parse({"test", "-a"}));
REQUIRE(config.abortAfter == 1);
}
SECTION("-x 2 aborts after two failures") {
CHECK(cli.parse({"test", "-x", "2"}));
REQUIRE(config.abortAfter == 2);
}
SECTION("-x must be numeric") {
auto result = cli.parse({"test", "-x", "oops"});
CHECK(!result);
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("convert") && ContainsSubstring("oops"));
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
Add command line option 'never' to --wait-for-keypress (#1866) Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2020-02-15 20:42:57 +01:00
SECTION("wait-for-keypress") {
SECTION("Accepted options") {
using tuple_type = std::tuple<char const*, Catch::WaitForKeypress::When>;
auto input = GENERATE(table<char const*, Catch::WaitForKeypress::When>({
tuple_type{"never", Catch::WaitForKeypress::Never},
tuple_type{"start", Catch::WaitForKeypress::BeforeStart},
tuple_type{"exit", Catch::WaitForKeypress::BeforeExit},
tuple_type{"both", Catch::WaitForKeypress::BeforeStartAndExit},
}));
CHECK(cli.parse({"test", "--wait-for-keypress", std::get<0>(input)}));
REQUIRE(config.waitForKeypress == std::get<1>(input));
}
SECTION("invalid options are reported") {
auto result = cli.parse({"test", "--wait-for-keypress", "sometimes"});
CHECK(!result);
#ifndef CATCH_CONFIG_DISABLE_MATCHERS
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("never") && ContainsSubstring("both"));
Add command line option 'never' to --wait-for-keypress (#1866) Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2020-02-15 20:42:57 +01:00
#endif
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
Add command line option 'never' to --wait-for-keypress (#1866) Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
2020-02-15 20:42:57 +01:00
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
SECTION("nothrow") {
SECTION("-e") {
CHECK(cli.parse({"test", "-e"}));
REQUIRE(config.noThrow);
}
SECTION("--nothrow") {
CHECK(cli.parse({"test", "--nothrow"}));
REQUIRE(config.noThrow);
}
}
SECTION("output filename") {
SECTION("-o filename") {
CHECK(cli.parse({"test", "-o", "filename.ext"}));
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
REQUIRE(config.defaultOutputFilename == "filename.ext");
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
SECTION("--out") {
CHECK(cli.parse({"test", "--out", "filename.ext"}));
Add support for multiple parallel reporters This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
2021-02-06 20:12:07 +01:00
REQUIRE(config.defaultOutputFilename == "filename.ext");
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
}
SECTION("combinations") {
SECTION("Single character flags can be combined") {
CHECK(cli.parse({"test", "-abe"}));
CHECK(config.abortAfter == 1);
CHECK(config.shouldDebugBreak);
CHECK(config.noThrow == true);
}
}
SECTION( "use-colour") {
using Catch::UseColour;
SECTION( "without option" ) {
CHECK(cli.parse({"test"}));
REQUIRE( config.useColour == UseColour::Auto );
}
SECTION( "auto" ) {
CHECK(cli.parse({"test", "--use-colour", "auto"}));
REQUIRE( config.useColour == UseColour::Auto );
}
SECTION( "yes" ) {
CHECK(cli.parse({"test", "--use-colour", "yes"}));
REQUIRE( config.useColour == UseColour::Yes );
}
SECTION( "no" ) {
CHECK(cli.parse({"test", "--use-colour", "no"}));
REQUIRE( config.useColour == UseColour::No );
}
SECTION( "error" ) {
auto result = cli.parse({"test", "--use-colour", "wrong"});
CHECK( !result );
Rename Contains string matcher builder to ContainsSubstring The problem with the old name was that it collided with the range matcher `Contains`, and it was not really possible to disambiguate them just with argument types. Closes #2131
2021-09-23 23:28:59 +02:00
CHECK_THAT( result.errorMessage(), ContainsSubstring( "colour mode must be one of" ) );
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
}
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
SECTION("Benchmark options") {
SECTION("samples") {
CHECK(cli.parse({ "test", "--benchmark-samples=200" }));
REQUIRE(config.benchmarkSamples == 200);
}
Hidden tests now require positive filter match to be selected This also required some refactoring of how the pattern matching works. This means that the concepts of include and exclude patterns are no longer unified, with exclusion patterns working as just negation of an inclusion patterns (which led to including hidden tags by default, as they did not match the exclusion), but rather both include and exclude patterns are handled separately. The new logic is that given a filter and a test case, the test case must match _all_ include patterns and _no_ exclude patterns to be included by the filter. Furthermore, if the test case is hidden, then the filter must have at least one include pattern for the test case to be used. Closes #1184
2019-06-22 20:26:03 +02:00
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
SECTION("resamples") {
CHECK(cli.parse({ "test", "--benchmark-resamples=20000" }));
REQUIRE(config.benchmarkResamples == 20000);
}
Allow configuring of benchmark warmup time
2020-01-27 15:43:27 +01:00
SECTION("confidence-interval") {
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
CHECK(cli.parse({ "test", "--benchmark-confidence-interval=0.99" }));
Move Approx out of the Detail namespace
2020-02-03 15:07:59 +01:00
REQUIRE(config.benchmarkConfidenceInterval == Catch::Approx(0.99));
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
}
Allow configuring of benchmark warmup time
2020-01-27 15:43:27 +01:00
SECTION("no-analysis") {
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
CHECK(cli.parse({ "test", "--benchmark-no-analysis" }));
REQUIRE(config.benchmarkNoAnalysis);
}
Allow configuring of benchmark warmup time
2020-01-27 15:43:27 +01:00
SECTION("warmup-time") {
CHECK(cli.parse({ "test", "--benchmark-warmup-time=10" }));
REQUIRE(config.benchmarkWarmupTime == 10);
}
Integrate Nonius benchmark into Catch2 Changes done to Nonius: * Moved things into "Catch::Benchmark" namespace * Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros * Removed Nonius's parameters for benchmarks, Generators should be used instead * Added relevant methods to the reporter interface (default-implemented, to avoid breaking existing 3rd party reporters) * Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default on MSVC * Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of benchmarking from Catch
2019-04-23 23:41:13 +02:00
}
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
}
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
TEST_CASE("Parsing sharding-related cli flags", "[sharding]") {
using namespace Catch::Matchers;
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
Catch::ConfigData config;
auto cli = Catch::makeCommandLineParser(config);
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
SECTION("shard-count") {
CHECK(cli.parse({ "test", "--shard-count=8" }));
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
REQUIRE(config.shardCount == 8);
}
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
SECTION("Negative shard count reports error") {
auto result = cli.parse({ "test", "--shard-count=-1" });
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
CHECK_FALSE(result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Shard count must be a positive number"));
}
SECTION("Zero shard count reports error") {
auto result = cli.parse({ "test", "--shard-count=0" });
CHECK_FALSE(result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Shard count must be a positive number"));
}
SECTION("shard-index") {
CHECK(cli.parse({ "test", "--shard-index=2" }));
REQUIRE(config.shardIndex == 2);
Allow test sharding for e.g. Bazel test sharding feature This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
2021-07-11 21:46:05 +02:00
}
Improve shardIndex/Count cli argument parsing
2021-10-26 23:26:07 +02:00
SECTION("Negative shard index reports error") {
auto result = cli.parse({ "test", "--shard-index=-12" });
CHECK_FALSE(result);
REQUIRE_THAT(result.errorMessage(), ContainsSubstring("Shard index must be a non-negative number"));
}
SECTION("Shard index 0 is accepted") {
CHECK(cli.parse({ "test", "--shard-index=0" }));
REQUIRE(config.shardIndex == 0);
}
All tests files have .tests.cpp suffix. Also moved tests out of TestMain.cpp and moved up a level
2017-11-13 17:03:27 +01:00
}
Add tests for escaping special characters in CLI
2019-09-21 00:11:45 +02:00
Allow enabling multiple warnings in one invocation
2021-12-18 20:48:43 +01:00
TEST_CASE( "Parsing warnings", "[cli][warnings]" ) {
using Catch::WarnAbout;
Catch::ConfigData config;
auto cli = Catch::makeCommandLineParser( config );
SECTION( "NoAssertions" ) {
REQUIRE(cli.parse( { "test", "-w", "NoAssertions" } ));
REQUIRE( config.warnings == WarnAbout::NoAssertions );
}
SECTION( "NoTests is no longer supported" ) {
REQUIRE_FALSE(cli.parse( { "test", "-w", "NoTests" } ));
}
SECTION( "Combining multiple warnings" ) {
REQUIRE( cli.parse( { "test",
"--warn", "NoAssertions",
"--warn", "UnmatchedTestSpec" } ) );
REQUIRE( config.warnings == ( WarnAbout::NoAssertions | WarnAbout::UnmatchedTestSpec ) );
}
}
Add tests for escaping special characters in CLI
2019-09-21 00:11:45 +02:00
TEST_CASE("Test with special, characters \"in name", "[cli][regression]") {
// This test case succeeds if we can invoke it from the CLI
SUCCEED();
}
Improve error handling in parsing reporter specs
2021-12-30 23:50:40 +01:00
TEST_CASE("Various suspicious reporter specs are rejected",
"[cli][reporter-spec][approvals]") {
Catch::ConfigData config;
auto cli = Catch::makeCommandLineParser( config );
auto spec = GENERATE( as<std::string>{},
"",
"::console",
"console::",
"console::some-file::",
"::console::some-file::" );
CAPTURE( spec );
auto result = cli.parse( { "test", "--reporter", spec } );
REQUIRE_FALSE( result );
}
Reference in New Issue Copy Permalink