As part of `-Wdeprecated-copy-dtor` sweep, I noticed that Capturer's
copies are defaulted. Given that the class would likely break horribly
in the event of actual copy happening, they are now deleted.
This should improve the compilation times by decreasing the number
of TUs compiled, without making overly big TUs that would cause
problems with heavy-tailed compilation times.
There is one "combined TU" for the top level part, and each subpart,
except for Reporters, which currently do not have any trivial TUs.
Special member functions are now implicit, which should make them
both noexcept and constexpr. The `operator<<` has been made into
hidden friend as per best practices.
As Clara is no longer maintained as a separate project, the
implementation was moved to the internal subfolder of top-level
folder. This removes one folder and avoids potential user confusion.
Also simplified the convenience header checking script accordingly.
There are two reasons for this:
1) It is highly unlikely that someone has use for this header,
which has no customization points and only provides simplest
possible main, and cannot link the static library which also
provides a default main implementation.
2) It being a header was causing extra complications with
the convenience headers, and our checking script. This would either
require special handling in the checking script, or would break user's
of the main convenience header.
All in all, it is simpler and better in the long term to remove it,
than to fix its problems.
I do not think we need a safeguard against not including files in
CMake anymore, and as it is, it caused annoying false positive about
the default main implementation.
At some point we moved over to catch2:catchorg (notice lowercase `c`)
instead of Catch2:catchorg, but we kept uploading the released
packages to the upper-cased repository... Time to fix this, and then
merge them again.
The naming scheme is simple, to include all matchers, include header
`catch2/matchers/catch_matchers_all.hpp`. To include **everything**,
include `catch2/catch_all.hpp`.
This describes the reality better, as it also links in the rest
of Catch2.
The on-disk name of the static library remains just `Catch2Main`,
as that is what it is -- single main function -- and on-disk artifacts
cannot describe link dependencies.
It is no longer used by v3, and contains obsolete versions of the
headers anyway. There are future plans for some sort of replacement,
but those are,
1) in the future
2) different than the single header model
so we can delete the folder completely.
Originally the tests were from #1912, but as it turned out, the issue
was somewhere else. Still, the inputs provided were interesting, so
they are now part of our test suite.
Catch assumes std::uncaught_exceptions is available whenever C++17 is
available, but for macOS versions older than 10.12 this is not the case.
Instead of checking the C++ version, use a macro to check whether the
feature is available.
Previously a random test ordering was obtained by applying std::shuffle
to the tests in declaration order. This has two problems:
- It depends on the declaration order, so the order in which the tests
will be run will be platform-specific.
- When trying to debug accidental inter-test dependencies, it is helpful
to be able to find a minimal subset of tests which exhibits the issue.
However, any change to the set of tests being run will completely
change the test ordering, making it difficult or impossible to reduce
the set of tests being run in any reasonably efficient manner.
Therefore, change the randomization approach to resolve both these
issues.
Generate a random value based on the user-provided RNG seed. Convert
every test case to an integer by hashing a combination of that value
with the test name. Sort the test cases by this integer.
The test names and RNG are platform-independent, so this should be
consistent across platforms. Also, removing one test does not change
the integer value associated with the remaining tests, so they remain in
the same order.
To hash, use the FNV-1a hash, except with the basis being our randomly
selected value rather than the fixed basis set in the algorithm. Cannot
use std::hash, because it is important that the result be
platform-independent.
It did not clear out all of its internal state when switching from
one pattern to another, so when it should've escaped `,`, it took
its position from its position in the original user-provided string,
rather than its position in the current pattern.
Fixes#1905
CATCH_INTERNAL_IGNORE_BUT_WARN() introduced with b7b346c triggers
clang-tidy warning 'cppcoreguidelines-pro-type-vararg' for every usage
of assertion macros like CHECK() and REQUIRE(). Silence it via NOLINT
in the '#if defined(__clang__)' block only, as clang-tidy honors those.
The old code caused warnings to fire under MSVC, and Clang <3.8.
I could not find a GCC version where it worked, but I assume that it
did at some point.
This new code causes all of MSVC, GCC, Clang, in current versions,
to emit signed/unsigned comparison warning in test like this:
```cpp
TEST_CASE() {
int32_t i = -1;
uint32_t j = 1;
REQUIRE(i != j);
}
```
Where previously only MSVC would emit the warning.
Fixes#1880
These files are not included by the default
`#include <catch2/catch_test_macros.hpp>` path, so that users do
not have to pay for them if they do not use them. Follow up is to
split out the small part of `catch_preprocessor.hpp` used by the
default test macros (AFAIK, it is just `INTERNAL_CATCH_REMOVE_PARENS`
macro), so that it is not included by the default path either.
Also fixes#1892 by providing the missing macros.
This is both a really big and a really small commit. It is small in
that it only contains renaming, moving and modification of include
directives caused by this.
It is really big in the obvious way of touching something like 200
files.
The new rules for naming files is simple: headers use the `.hpp`
extension. The rules for physical file layout is still kinda in
progress, but the basics are also simple:
* Significant parts of functionality get their own subfolder
* Benchmarking is in `catch2/benchmark`
* Matchers are in `catch2/matchers`
* Generators are in `catch2/generators`
* Reporters are in `catch2/reporters`
* Baseline testing facilities are in `catch2/`
* Various top level folders also contain `internal` subfolder,
with files that users probably do not want to include directly,
at least not until they have to write something like their own
reporter.
* The exact files in these subfolders is likely to change later
on
Note that while some includes were cleaned up in this commit, it
is only the low hanging fruit and further cleanup using automatic
tooling will happen later.
Also note that various include guards, copyright notices and file
headers will also be standardized later, rather than in this commit.
It used to be a file that would collect interfaces we always wanted
to provide to users, so that the single header stitching script
would place them in the common part of the single header version.
As v3 is moving to separate headers model, the file is no longer
useful.
This was an old "include all" header, that we no longer want to be
usable, to make the include differences in new versions explicit.
We will introduce new "include all" headers later, in the form of
`catch_all.hpp`, `catch_matchers_all.hpp` and so on...
Removed nested `StdString` namespace and added Doxygen comments.
Also renamed some matchers to avoid colisions now that there are
less separate namespaces for matchers to go to. Since this is a
breaking release anyway, it shouldn't matter, and the factory
functions that the users should use remain the same anyway.
Removed the `generic` nested namespace, so PredicateMatcher now
lives in `Catch::Matchers` namespace, just like other matchers.
Also cleaned up and doxygenized comments on the `Predicate` factory
function for `PredicateMatcher`.
The two changes are
`catch_matchers_templates` -> `catch_matchers_templated` and
`catch_matchers_generic` -> `catch_matchers_predicate`. The former
is mostly cosmetic, but the second was previously significantly
misleading, and as the library is now to be consumed by including
specific headers, this needed to be fixed.
`SizeIs` can accept both `size_t` and a matcher. In the first case,
it checks whether the size of the range is equal to specified size.
In the second case, it checks whether the provided matcher accepts
the size of the range.
Outside of `MatcherBase` and `GenericMatcherBase`, matchers are not
designed to be overriden. This means that doing so can easily lead
to errors, and matchers are generally fairly simple functionality-wise.
so there is not much code reuse to be gained anyway.
Thus, Catch2-provided concrete matchers are now final.
In general, for Catch2 v3 we are making virtual types `final`,
unless they were explicitly designed to be derived-from.
`ListeningReporter` is definitely not designed to be derived-from.
This was previously used to avoid `dynamic_cast` inside our code,
when we were creating more than one reporter, or a reporter
together with listeners. However, since then the offending code
was refactored to be smarter instead, and this query member function
is no longer needed nor used.
The only way to stream those is to use the `bool` overload of `op<<`.
However, to convert a function to bool, GCC creates AST equivalent
of `A? true : false`. Then, because `A` is a function, it warns that
it will never be `false`. 🤦
As a bonus, newer GCC versions issue _two_ different warnings about
this, but older GCC versions do not know both of them, so we also
have to suppress warning about unknown warning suppression.
The variable initialization has test registration as a side-effect,
but as far as GCC is concerned, the variable itself is unused.
Because the macro substitution always happens at global scope, we
cannot use cast to `void` as is usually done.
Sadly most versions still cannot properly handle the suppression
via `_Pragma`, so it has to leak to the users when they use older
GCC versions to compile their code
In the future we can expect many more matchers, so let's give them
a place to live.
Also moved matcher-related internal files to `internal` subfolder.
Ideally we should sort out all of our source code, but that will
have to come later.
This commit also forbids composing lvalues of composed matchers, as
per previous deprecation notice. I do not expect this to be contentious
in practice, because there was a bug in that usage for years, and
nobody complained.
Thanks to the changes to compilation model, the tests for benchmarking
macros have been made part of the normal test run. This means that
the only purpose these separately compiled tests served was to waste
CI time.
Given that in the 2 or so years that matchers are thing nobody complained,
it seems that people do not actually write this sort of code, and the
possibility will be removed in v3. However, to avoid correctness bugs,
we will have to support this weird code in v2.
C++11 math requires _GLIBCXX_USE_C99_MATH_TR1 to be true with gcc/clang.
Also fixes an issue with uClibc-ng where __UCLIBC__ is defined in features.h but
that is not included here and is thus no-op.
- Overrides added
- usages of push_back() replaced with emplace_back()
- Loop variable made const-refernce
- NULL replaced with nullptr
- Names used in the declaration and definition unified
- size() replaced with empty
- Identical cases merged
This commit extends the Matchers feature with the ability to have type-independent (e.g. templated) matchers. This is done by adding a new base type that Matchers can extend, `MatcherGenericBase`, and overloads of operators `!`, `&&` and `||` that handle matchers extending `MatcherGenericBase` in a special manner.
These new matchers can also take their arguments as values and non-const references.
Closes#1307Closes#1553Closes#1554
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
b77cec05c0 fixed this problem for tagging tests, so that a test
case tagged with `[.foo]` would be parsed as tagged with `[.][foo]`.
This does the same for the test spec parsing.
Fixes#1798
Copying a `ReusableStringStream` would lead to "double free" of
the stream, and thus it could be used in multiple places at the
same time, breaking the output.
On systems where std::chrono::steady_clock::period is not std::nano, benchmark tests fail to compile due to trying to convert analysis.samples from a vector of duration<double, clock::period> to a vector of std::chrono::duration<double, std::nano>.
Its intent was to show which headers are expected to be useable by
Catch2's users, and to enforce their inclusion in the single header
distribution at the right place.
Given the new library model, the second use case is not needed and
the first one is better served with documentation and physical file
layout.
Now that Catch2 is a proper library, we can always build the full
library (comparatively minor slowdown) and the user can avoid
including benchmarking headers to avoid the compilation slowdown.
When running tests in parallel, CTest runs the tests in decreasing
order of cost (time required), to get the largest speed up from
parallelism. However, the initial cost estimates for all tests are
0, and they are only updated after a test run. This works on a dev
machine, where the tests are ran over and over again, because
eventually the estimates become quite precise, but CI always does
a clean build with 0 estimates.
Because we have 2 slow tests, we want them to run first to avoid
losing parallelism. To do this, we provide them with a cost estimate
manually.
Previously, we would collect coverage data for all source files in
Catch2's directory, including tests and examples, and we would then
ask codecov.io to ignore those. With this change, OpenCppCoverage
only collects coverage data for source files in the `src/` directory.
This cuts the size of the coverage report in half, and also speeds
up the coverage collection.
The use we previously used the polyfill or naked new is that we
supported C++11, which did not yet have `std::make_unique`. However,
with the move to C++14 as the minimum, `std::make_unique` can be
expected to be always available.
Because some of the tooling used by Catch2 does not properly support
version postfixes, such as `preview-1`, we will report the
in-development version is `v3.0.0`, and the first real release will
have to be `v3.0.1`.
Closes#1824
Now that the recommended distribution and usage method is proper
library, users can just avoid including the matcher headers to get
basically the same effect.
This should decrease the number of allocations before main is entered
significantly, but complicates the code somewhat in return.
Assuming I used `massif` right, doing just `SelfTest --list-tests`
went from 929 allocations at "Remove gcc-4.9 from the travis builds"
(2 commits up), to 614 allocations with this commit.
Now a `TEST_CASE` macro should create a single TestCaseInfo and then
it should never be copied around. This, together with latter changes,
should significantly decrease the number of allocations made before
`main` is even entered.
According to CMake, it does not support templated variables. Strictly
speaking, we added that to the target so that the project defaults
to C++14, but I am planning to use them inside Catch2 anyway.
Users can still write a description for their sections, but it will
no longer be saved as part of the `SectionInfo` struct. This ability
has also been added to the documentation.
Closes#1319
* Use Xenial as the base distribution
* Remove C++11 builds
* Add a lot more C++14 builds
* Add some C++17 builds
* Include newer versions of Clang and GCC
This also required some refactoring of how the pattern matching
works. This means that the concepts of include and exclude patterns
are no longer unified, with exclusion patterns working as just
negation of an inclusion patterns (which led to including hidden
tags by default, as they did not match the exclusion), but rather
both include and exclude patterns are handled separately.
The new logic is that given a filter and a test case, the test
case must match _all_ include patterns and _no_ exclude patterns
to be included by the filter. Furthermore, if the test case is
hidden, then the filter must have at least one include pattern
for the test case to be used.
Closes#1184
This allows us to provide machine-readable listings through the
XMLReporter and it will also allow users to define their own listing
format that does whatever their own tools need.
Previously it returned the sum of listed things because ???. This
was completely useless and in many ways actively counterproductive
because of the success/failure conventions around exit codes.
Closes#1410
Previously, each warning suppression was self-contained, with its
own pair of `SUPPRESS_X_WARNING` and `UNSUPPRESS_X_WARNING` macros.
This had the obvious advantage of being self-containing, but it
also meant that if we needed to suppress more than one warning
in a single place, then we would manipulate the compiler's warning
state multiple times, even though logically we would only need one
layer.
The new way of suppressing warnings in macros is to push compiler's
warning state with `CATCH_INTERNAL_START_WARNINGS_SUPPRESSION` macro,
then disable whatever macros we need with the
`CATCH_INTERNAL_SUPPRESS_X_WARNINGS` macro, and then return to the
previous state using `CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION`.
The JUnit report is improved in that:
* The message shows the testing condition, not the result
* The actual message has similar output than the console one
Now it no longer tries to be this weird hybrid between an owning
and non-owning reference, and is only ever non-owning. This is also
reflected in its interface, for example `StringRef::isNullTerminated`
is now public, and `StringRef::c_str()` has the precondition that it
is true.
Overview of the changes:
* The `StringRef::m_data` member has been completely removed, as it
had no more uses.
* `StringRef::isSubstring()` has been made public and renamed to
`StringRef::isNullTerminated()`, so that the name reflects what the
method actually does.
* `StringRef::currentData()` has been renamed to `StringRef::data()`,
to be in line with common C++ containers and container-alikes.
* `StringRef::c_str()` will no longer silently make copies. It instead
has a precondition that `isNullTerminated()` is true.
* If the user needs a null-terminated string, they should use the
`std::string` conversion operator and call `c_str()` on the resulting
`std::string`.
* Some small optimizations in various places.
* Basic functionality is now `constexpr`.
Unless someone steps up to fix the long link times with a set of
unobtrusive changes, the recommended solution will remain "use a better
linker".
Related to #1205, #1247, and #1637Closes#1247Closes#1637
This should now properly handle small numbers which would previously
output something like `[0.00000000000000019, 0.00000000000000019]`,
which does not allow user to read the numbers properly.
Closes#1760
this warning was introduced by rework to support NTTPs
since we have implementation macro for NTTPs and normal template test cases
warning is going to be suppressed
Fixes#1762
Only works for exceptions that publicly derive from `std::exception`
and the matching is done exactly, including case and whitespace.
Closes#1649Closes#1728
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Your branch is up-to-date with 'origin/master'.
#
# Changes to be committed:
# modified: ../docs/matchers.md
# modified: ../include/internal/catch_capture_matchers.h
# modified: ../projects/CMakeLists.txt
# modified: ../projects/SelfTest/Baselines/compact.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/console.std.approved.txt
# modified: ../projects/SelfTest/Baselines/console.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/junit.sw.approved.txt
# modified: ../projects/SelfTest/Baselines/xml.sw.approved.txt
# modified: ../projects/SelfTest/UsageTests/Matchers.tests.cpp
#
# Untracked files:
# ./
# ../clang-full/
# ../clang-test/
# ../clang10-build/
# ../coverage-build/
# ../gcc-build/
# ../gcc-full/
# ../include/internal/catch_matchers_exception.cpp
# ../include/internal/catch_matchers_exception.hpp
# ../misc-build/
# ../msvc-sln/
# ../notes.txt
# ../test-install/
#
It checks Knuth's _close enough with tolerance_ relationship, that
is `|lhs - rhs| <= epsilon * max(|lhs|, |rhs|)`, rather then the
_very close with tolerance_ relationship that can be written down as
`|lhs - rhs| <= epsilon * min(|lhs|, |rhs|)`.
This is because it is the more common model around the internet, and
as such is likely to be less surprising to the users. In the future
we might want to provide the other model as well.
Closes#1746
This means that if you nest multiple random generators inside one
test case, they will not return the same sequence of numbers.
Idea taken from #1736 by Amit Herman.
Closes#1736Closes#1734
In the future, we will also want to introduce our own
`uniform_int_distribution` and `uniform_real_distribution` to get
repeatable test runs across different platforms.
Wrong nesting of namespaces resulted in the `Catch` namespace
being ambigous between `::Catch` and `::{anon}::Catch` namespaces.
This should fix it.
Closes#1761
Instead, let it be installed as a dependency of `conan-package-tools`
to avoid trouble with the fact that pip is really bad at version
resolution, and that up-to-date version of the `conan` package is not
supported by up-to-date version of the `conan-package-tools` package.
It was used in checking that types in TEMPLATE_TEST_CASE and friends
were unique, but this was removed for v2.8.0 (#1628). Since there
are no further uses of this trait, the simplest thing to do is to
just remove it.
Fixes#1757
The leading/trailing whitespace is problematic because of e.g.
`WHEN` macro having preceeding whitespace for alignment, and it is
generally messy.
Credits to Phil who did lot of the original work.
Closes#1708
This way it is explicit when there is a `StringRef` -> `std::string`
conversion and makes it easier to look for allocations that could
be avoided.
Doing this has already removed one allocation per registered test
case, as there was a completely pointless `StringRef` -> `std::string`
conversion when parsing tags of a test case.
The old code was a left-over from the times when the
`capturedExpression` member was a `const char*`, which could always
be indexed. With the change to use `StringRef`, blindly indexing 0th
element is invalid, as it is not indexable part of a StringRef.
The parameter given to `convert` may not be copyable therefore it has to be
captured by const reference. For example an `std::tuple` that contains a
non-copyable type is itself non-copyable.
The NonDefaultConstructible test-case was reduced by one example type
because it did not add any value.
`print` version of the logging functions supports `printf`-like
formatting, which we do not use and given our current debug print
internals, will never use. This should be slightly more efficient
and expresses the intent better.
They lead to stringification of file (which is ok) and file line
(not ok) to the approvals, which makes them exceedingly brittle
and not worth approval testing. Instead we just run them as part
of the base test run.
This PR ultimately does 3 things
* Separately tracks matched tests per each filter part (that is, a set of filters separated by an OR (`,`)), which allows Catch2 to report each of the alternative filters that don't match any tests.
* Fixes `-w NoTests` to return non-zero in the process
* Adds tests for `-w NoTests`.
Noticed that the code was originally concatenating strings just to
then append the result to another string. Now it does not create
temporaries and also preallocates the string buffer.
Under WSL, Python in text mode will translate `\n` into `\r\n`, even
though other tools and utilities use `\n` (because WSL is basically
Linux). This leads to the update scripts leaving the files with
Windows newlines even though git and similar expect them to have
Linux newlines.
By instead handling files in binary mode, we can keep the original
newlines. This commits switches parts of the update process to
binary mode, but not all because some of the will require a lot of
work to fix.
Last time it was fixed to a specific version because the `conan`
and the `conan-package-tools` package that `pip install` would
gather were not compatible, let's hope it won't happen again.
* Fix non-default-constructible type lists used in TEMPLATE_LIST_TEST_CASE
std::tuple is not default constructible when the first type is not
default-constuctible. Therefore it can not be instantiated.
to circumvent this, we have to use std::declval in the unevaluate decltype
context.
For some time now (I'd guess almost a year 🤷), the coverage
merging on Windows has been failing, because the reports have been
generated in a different folder than expected. Our merge script did
not report failure because it was not checking the returned error
code from OpenCppCoverage, and for some reason, the `codecov` tool
happily returned 0 even though it did not find the file it was
supposed to upload...
The former is also fixed by this commit.
Parsing --list-tests is broken, as Catch automatically line wraps the
line when it gets too long, stripping any whitespace in the process.
This means that it's impossible to reproduce the exact name of the
test if the test's name is long enough to line-wrap.
Furthermore, overwriting the LABELS property with the discovered labels
breaks users who manually added custom ctest labels.
Rolling back to using --list-test-names-only for now, as it does not
wrap lines even on very long test names.
We may be able parse the output of --list-tags to produce the ctest labels.
However, the straightforward way of doing this is to use CMake's
get_property(TEST ...) and set_property(TEST ... APPEND ...), which don't
work if the test name has spaces or other special characters. We would
need to mangle the test name to a valid CMake identifier to do it that way.
- The current setup tries to detect USE_CPP14/USE_CPP17 and sets the
CXX_STANDARD property for the SelfTest target. This is not ideal, since
CMAKE_CXX_STANDARD can be provided by the toolchain file or as command line
option and should be used by the library internally correctly. Hence, the
whole set of the relevant lines from `projects/CMakeLists.txt` have been
removed.
- The above can also cause subtle issues where the user is expecting the tests
to compile with C++17 after setting CMAKE_CXX_STANDARD and then getting
results of compilation with C++11 as USE_CPP17 has not been set.
- The current build matrix used the above code to run the tests. So, even
though the it should not required anymore to build Catch2, it was still
required to send correct options to build matrix. In that respect,
.travis.yml has been modified to send correct options to the build command
in the new setup.
This keeps it out of the main include path when benchmarking is
enabled, somewhat reducing the compilation-time penalty.
Also moved some other functions into the .cpp file, especially
helpers that could be given internal linkage, and concretized some
iterator-templated code that only ever used
`std::vector<double>::iterator`.
This allows us to move <stdexcept> out of the common path, and replace
it with just <exception>. The difference between these two headers is
~13k lines after preprocessing on libstdc++ (16k vs 3k) and ~17k lines
for MS's STL(33k vs 16k).
Note that this is only beneficial if no other stdlib header we use
includes <stdexcept>. AFAIK this is true for the newest MS's STL,
but I have no idea of the applicability for libstdc++ and libc++.
* Units from <ratio> are no longer redeclared in our own namespace
* The default clock is `steady_clock`, not `high_resolution_clock`,
because, as HH says "high_resolution_clock is useless. If you want
measure the passing of time, use steady_clock. If you want user
friendly time, use system_clock".
* Benchmarking support is opt-in, not opt-out, to avoid the large
(~10%) compile time penalty.
* Benchmarking-related options in CLI are always present, to decrease
the amount of code that is only compiled conditionally and making
the whole shebang more maintainble.
Changes done to Nonius:
* Moved things into "Catch::Benchmark" namespace
* Benchmarks were integrated with `TEST_CASE`/`SECTION`/`GENERATE` macros
* Removed Nonius's parameters for benchmarks, Generators should be used instead
* Added relevant methods to the reporter interface (default-implemented, to avoid
breaking existing 3rd party reporters)
* Async processing is guarded with `_REENTRANT` macro for GCC/Clang, used by default
on MSVC
* Added a macro `CATCH_CONFIG_DISABLE_BENCHMARKING` that removes all traces of
benchmarking from Catch
This fixes an issue where a self-assignment of a StringRef copy would point into internally (and now dangling) data.
(now self-assignment check is no longer needed)
Previously we had them to avoid including <algorithm> in the vector
matchers, but
* we included it anyway, even though we did not use it
* we use <algorithm> anyways in the generators
@ThijsWithaar is responsible for giving me the idea, but his PR
had couple of things that meant it was simpler to rewrite it than
to fix and merge it.
Supersedes and closes#1599
Eventually this needs to be fixed in the textflow project by Phil,
but he has not done so in the half a year this bug has been known
to be there, so...
Closes#1470Closes#1455
After the script, the ParseAndAddCatchTests_TESTS property for the
target, and for each source file in the target is set, and contains the
list of the tests extracted from that target, or from that file.
This is useful, for example to add further labels or properties to the
tests.
* Deduce map return type implicitly
Giving the first template argument to map generator function to deduce
return type is now optional even if the return type is different from
the type generated by mapped generator.
This adds UNSCOPED_INFO macro, creating a log message that is stored
until the end of next assertion or the end of test case, whichever comes
first. These messages are not scoped locally, unlike messages created by
INFO macro.
This avoids the problem where writes to stderr/stdout stop being
line-buffered when stderr/stdout is redirected to a file, which led
to different order of outputs between Linux and Windows in our tests.
This generator collects values from the underlying generator until it
has a specified amount of them, and then returns them in one "chunk".
In case the underlying generator does not have enough elements for
a specific chunk, the left-over elements are discarded.
Closes#1538
7f229b4f caused the output file to get opened twice, while
some types of files (e.g. named pipes) can be only opened once.
After this change Session::applyCommandLine opens the output file
only when there is an error to print.
Previously, for a TEMPLATE_PRODUCT_TEST_CASE("Test" ..., T, (P1, P2)),
the generated test case names were
Test - 0
Test - 1
With this commit, the correct typename is used:
Test - T<P1>
Test - T<P2>
-----------
MSVC needs another indirection to evaluate INTERNAL_CATCH_STRINGIZE
and also inserts a space before theINTERNAL_CATCH_STRINGIZE_WITHOUT_PARENS
parameter, which we can get rid of by pointer arithmetic.
The REQUIRE_THROWS and CATCH_REQUIRE_THROWS macros have
a subtle inconsistency in their implementation which can
cause a warning if [-Wunused-value] is used.
This commit changes CATCH_REQUIRE_THROWS so it has the
same implementation as REQUIRE_THROWS
It looks like REQUIRE_THROWS was change in commit
fae0fa4ec but not CATCH_REQUIRE_THROWS.
Similar changes for CATCH_CHECK_THROWS
By using non-trivially copyable types, we force libstdc++-9's variant to
properly enter the valueless-by-exception state for our stringification
test.
Related to #1511
The old template that combined both bug reports and issue requests has led to various weird issues being filed, maybe having a separate one for each will help.
The clock estimator has a potential division by zero.
Using `iteration + 1` seems also more logical to me for
an average.
Found with coverity in a downstream project.
When PARSE_CATCH_TESTS_ADD_TARGET_IN_TEST_NAME is enabled the cmake helper script fails to extract the testcase name if a whitespace is before the name string. Use regex to consider and remove this whitespace.
fix by Mike-Devel
fixes: https://github.com/catchorg/Catch2/issues/1493
The ostream passed as reference to `hexEscapeChar` is manipulated
and its original state not restored. This fixes it.
Seen via coverity in a downstream project.
As explained in issue #1273, `operator&&` and `operator||` should give
a proper compile time error on use instead of the compiler complaining
about them not being defined. This commit adds an `always_false` type in
`catch_meta.hpp` used for implementing a nice `static_assert` for both
of the abovementioned operators.
Closes#1273
This doesn't cause trouble with GCC/Clang and libstdc++, but IAR
and its stdlib apparently doesn't compile when you use `fno-exceptions`
and `std::current_exception`/`std::rethrow_exception`.
Fixes#1462
support for generating test cases based on multiple template template
types combined with template arguments for each of the template template
types specified
e.g.
```
TEMPLATE_PRODUCT_TEST_CASE("template product","[template]",
(std::tuple, std::pair, std::map),
((int,float),(char,double),(int,char)))
```
will effectively create 9 test cases with types:
std::tuple<int,float>
std::tuple<char,double>
std::tuple<int,char>
std::pair<int,float>
std::pair<char, double>
std::pair<int,char>
std::map<int,float>
std::map<char,double>
std::map<int,char>
Tested type is accessible in test case body as TestType
Unique name is created by appending ` - <index>` to test name
since preprocessor has some limitations in recursions
Closes#1454
This captures the intent better, as some changes are indeed plain
deprecations leading to removal, but other changes can be viewed
as minor tune-ups instead.
The previous implemetation was just plain broken for most of
possible uses, the new one should work (even though it is ugly
as all hell, and should be improved ASAP).
Fixes#1436
If this option is enabled and PARSE_CATCH_TESTS_NO_HIDDEN_TESTS option is disabled, the test is be added, but the DISABLED property is set, therefore CTest shows it as "Not Run (Disabled)" instead of "Passed"
Since https://github.com/catchorg/Catch2/pull/1405 was merged and propagated to the single include declaring a user operator<< in the global namespace makes it available to Catch2 string converters.
This adds support for templated tests and test methods via
`TEMPLATE_TEST_CASE` and `TEMPLATE_TEST_CASE_METHOD` macros. These
work mostly just like their regular counterparts*, but take an
unlimited** number of types as their last arguments.
* Unlike the plain `TEST_CASE*` macros, the `TEMPLATE*` variants
require a tag string.
** In practice there is limit of about 300 types.
This is a temporarily workaround until we can nuke the current
verbosities system from the orbit and replace it with something
actually sane.
Fixes#1426
No matcher actually uses it, and there is no good reason for it,
as the best it can do for user is removing a single indirection
when using the pointer inside the matcher. Given the overhead of
other code that will be running during such time, it is completely
meaningless.
This also fixes compilation for PredicateMatcher<const char*>.
This variable is set to allow the use of the nice ParseAndAddCatchTests script
in the case where a launcher is needed to execute the script.
This is introduced to allow to launch unit tests using mpi. In this case one can
write for instance
set(OptionalCatchTestLauncher ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${NUMPROC})
before calling the ParseAndAddCatchTests function.
This fixes 3 problems:
* Relative paths on Windows are now supported
* Out-of-tree (paths starting with ../) builds are now supported
* Path separator normalization no longer affects non-path components of input (problem with Compact reporter)
Fixes#1379Fixes#1222Fixes#1200Fixes#1194
simple code with provided main function which just returns 0
leaks memory due to fact that singletons are not cleaned up
running valgrind on such simple application reports that 752 bytes
are still available in 11 blocks
this commit adds destructor to Catch::LeakDetector which calls
Catch::cleanUp()
By default, it expands into a `static_assert` + `SUCCEED` pair, but
it can also be deferred to runtime by defining
`CATCH_CONFIG_RUNTIME_STATIC_REQUIRE`, which causes it to expand
into plain old `REQUIRE`.
Closes#1362Closes#1356
* Session::applyCommandLine overload on wchar_t
This allows users on Windows to use Catch::Session::applyCommandLine
with wchar_t * arguments of application.
With this change Session::run became templated so both char and wchar_t
version have the same implementation.
Xml result of reported will now contain value of rng-seed in case it
is not zero.
The value will be stored in element Randomness and it's attribute seed.
Relates to #1402
Some platforms set the signedness of char to unsigned (eg. ARM).
Convert from char should not assume the signedness of char.
Fix build issue with -Werror,-Wtautological-unsigned-zero-compare flags.
Signed-off-by: Miguel Gaio <mgaio35@gmail.com>
Previously a mismatched prefix would be skipped before the actual
comparison would be performed. Obviously, it is supposed to be
_matching_ prefix that is skipped.
Prevent warnings
- gnu: -Wcomment: multi-line comment
- clang: -Wweak-vtables 'class' has no out-of-line virtual method definitions; its vtable will be emitted in every translation unit
- clang: -Winconsistent-missing-override: 'method' overrides a member function but is not marked 'override'
- MSVC: C4702: unreachable code
This fixes some wording that implies C++98 standard, updates
the recommended solution to looped SECTION macros and mentioned
the "last section failed, test needs to be rerun" problem.
Related to #1367
Related to #1384
Related to #1389
This might prove helpful when the package managers either doesn't
have Catch at all, or provides it in obsolete version (Ubuntu 16.04,
I am looking at you).
Closes#1383
The "percentage" suggests that the expected epsilon can be in
[0, 100], but the expected values are in [0, 1]. The new wording
uses "coefficient", to make it clearer that we are talking about
values in [0, 1].
Closes#1388
The StringMaker is off by default and can be enabled by a new macro `CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER`, to avoid increasing the footprint of stringification machinery by default.
The desired behaviour was to match a literal "[.]", so the regex
has to be escaped as "\\[\\.\\]" -- double backslashes, because
it has to be escaped from CMake as well as from the regex engine.
This means
* Adding new configuration toggle `CATCH_CONFIG_DISABLE_EXCEPTIONS`
and a best-guess configuration auto-checking for it.
* Adding new set of internal macros, `CATCH_TRY`, `CATCH_CATCH_ALL`
and `CATCH_CATCH_ANON` that can be used in place of regular `try`,
`catch(...)` and `catch(T const&)` respectively, while disappearing
when `CATCH_CONFIG_DISABLE_EXCEPTIONS` is enabled.
* Replacing all uses of `throw` with calls to `Catch::throw_exception`
customization point.
* Providing a default implementation for the above customization point
when `CATCH_CONFIG_DISABLE_EXCEPTIONS` is set.
* Letting users override this implementation with their own.
* Some minor changes and ifdefs all around to support the above
issue #1360
It is possible to have multple given contexts in a single BDD scenario;
if you have to type 'and' in the GIVEN description; it's very likely you
need an AND.
A generic AND is not possible, thus a AND_GIVEN is added to complement
the AND_WHEN and AND_THEN.
Can be used without needing to increase indent:
SCENARIO("...") {
GIVEN("...")
AND_GIVEN("...") {
WHEN("...") {
THEN("...") {
// ...
}
}
}
}
would correctly output, when requested/needed:
Given: ...
And given: ...
When: ...
Then: ...
The padding had to be increased by a character in the output message, to
continue to be uniform.
This is because otherwise the installations paths provided via
GNUInstallDirs become messed up and parts of the installation
package will end up in the wrong place.
Also it doesn't make much sense to force dependees to also install
our header alongside them.
Closes#1373
Because of a change in VS toolset, missing option <UseFullPaths>
is no longer interpreted as "don't pass /FC to the compiler", but
rather as "pass /FC to the compiler". This is problematic, because
/FC not only changes how much of the path is reporter by the compiler
(e.g. in `__FILE__` macro), but it also lower cases the path.
This lower-casing of the path broke our approval tests for VS2017
about 5 months ago.
Using CMake 3.13 (not yet released) would also let us fix it, but
for now we use a vcxproj.user file that is merged with the main project
and explicitly disables `/FC`.
This means
* a new cmake option, `CATCH_BUILD_EXTRA_TESTS`, that conditionally
includes the ExtraTests subfolder
* building and running them on some of the Travis build images
* An example configuration test
In the future these should be extended to cover most of the
configuration options in Catch2, but this is a start.
The support is to be considered experimental, that is, the interfaces,
the first party generators and helper functions can change or be removed
at any point in time.
Related to #850
In case of 2 instances of SourceLineInfo constructed in the same
file, they will have the same `file` pointer (even at O0). Thus, we
can check if they are equal before calling potentially pointless
`strcmp`.
In theory the copy is cheap (couple of pointers change), but tests
are usually compiled in Debug mode/with minimal optimizations, which
means that most users will still have to pay the cost for those
function calls.
Because the macro name is compile-time constant, we do not have to
worry about lifetimes and will avoid allocation in case of missing
SSO or long macro name.
By opting the JUnit and XML reporters into it, we no longer run
into problem where they underreport the results without `-s` flag.
Related to #1264, #1267, #1310
This also goes for pkg-config installed by our CMake installation.
This includes
* Updating CMake version on Travis
* Adding a `Catch2` subfolder to the `single_include/` folder to
provide this include path both _inside_ the repository, and _outside_.
* Updated examples to build with the new paths
* Other general CMake cleanup
- seems the #ifdef was necessary after all, because of the difference in the way the cpp files are included in the full project vs the single include
- in the OC project I moved the #include of catch_tostring.cpp first. That solves the project for now, but is a brittle solution
Happening when using clang and templated operators, clang cannot decide
between the operator provided by ReusableStringStream and the one provided
by the value value as both are templates. This is easily solved by calling
the operator<< through the member syntax.
Fixes#1285
This was removed in 64be2ad, to fix OS X approval tests. At the time
I couldn't investigate because I didn't have access to OS X, but this
fixed it (and since we don't have MinGW in CI, the breakage went
unnoticed).
As it turns out, piece-wise compilation of the Compact
reporter had broken OS X detection for a long time, and fixing it
was what broke the approvals. After the approval scripts were
changed to compensate, this change passes approval tests and fixes
Until now, the stack size for POSIX signal handling was determined by
the implementation defined limit `STKSZ`, which in some cases turned out
to be insufficient, leading to stack overflow inside the signal handler.
The new size, which was determined experimentally, is the larger of 32kb
or `MINSTKSZ`.
Fixes#1225
This should align more closely with the intended semantics, where
types without `StringMaker` specialization or `operator<<` overload
are passed down to the user defined fallback stringifier.
Related to #1024
This support is based on overriden `std::exception::what` method, so
if an exception does not do so meaningfully, the message is still
pointless.
This is only used as a fallback, both `StringMaker` specialization and
`operator<<` overload have priority..
Theoretically the previous was not a valid YAML at all, but it is
fairly common for parsers to accept it, just in a wrong way. This
results in a configuration where only the last value for duplicate
keys is taken, instead of a hard error.
Android apparently does not support `std::to_string`, so we add a
small polyfill over it. Right now only the ULP matcher uses it,
but we have had plans to use it in `StringMaker<int>` and friends,
as it performs a lot better than `std::stringstream` based
stringification on MSVC.
See #1280 for more details
The commands provided have to be executed in the current gdb/lldb session or copied
into the users ~/.gdbinit ~/.lldbinit files to permanently skip debugging Catch code.
Fixes#904
While the comment format was valid C++, it breaks our tooling badly.
I opened up a github issue for our tooling, because unexpected
formatting of a block comment should not silently generate invalid
single header file, see #1269.
Unlike the relatively non-invasive old way of capturing stdout/stderr,
this new way is also able to capture output from C's stdlib functions
such as `printf`. This is done by redirecting stdout and stderr file
descriptors to a file, and then reading this file back.
This approach has two sizeable drawbacks:
1) Performance, obviously. Previously an installed capture made the
program run faster (as long as it was then discarded), because a call
to `std::cout` did not result in text output to the console. This new
capture method in fact forces disk IO. While it is likely that any
modern OS will keep this file in memory-cache and might never actually
issue the IO to the backing storage, it is still a possibility and
calls to the file system are not free.
2) Nonportability. While POSIX is usually assumed portable, and this
implementation relies only on a very common parts of it, it is no
longer standard C++ (or just plain C) and thus might not be available
on some obscure platforms. Different C libs might also implement the
relevant functions in a less-than-useful ways (e.g. MS's `tmpfile`
generates a temp file inside system folder, so it will not work
without elevated privileges and thus is useless).
These two drawbacks mean that, at least for now, the new capture is
opt-in. To opt-in, `CATCH_CONFIG_EXPERIMENTAL_REDIRECT` needs to be
defined in the implementation file.
Closes#1243
std::isalnum expects an int in the range of unsigned char or -1 (EOF),
otherwise it exhibits undefined behavior. Casting from char to unsigned
char avoids this.
MSVC warns about this when compiling with /analyze.
Catch2's documentation promises that listeners are called _before_
reporters, but because of the previous implementation, they were
called _after_ reporters. This commit fixes that.
Closes#1234
The fix leaves an open question: should we keep treating refs
to static array of chars as strings, or should we instead
use `strnlen` to check if it is null-terminated within the buffer
Fixes#1238
In CMake module both include and include/catch are added includes
lookup path. Examples are built with #include "catch.hpp" not
#include "catch/catch.hpp". This should be the same with pkg-config.
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Specific platforms (e.g. TDM-GCC) can have terrible timer resolution,
and our checking code will then loop for an inordinate amount of time.
This change will make it so that the calibration gives up after 3
seconds and just uses the already measured values.
This leaves one open question, how to signal that the resolution
is terrible and benchmarking should not happen?
Fixes#1237
There are still some holes, e.g. we leave surrogate pairs be
even though they are not a part of valid UTF-8, but this might
be for the better -- WTF-8 does support surrogate pairs inside
text.
Closes#1207
VS 2017 has an annoying bug, where the result of `__FILE__`
substitution is always lower-cased. This breaks approval tests
and I am not quite convinced that we should fully normalized paths
to accomodate this bug.
We need to remember to undo this in the future though.
DJGPP cross compiler is targeting DOS which does not support POSIX
signals. Probably for the same reason (targeting DOS) this compiler
does not support wide characters.
The old version would lead to error when Catch was installed
as a subproject. The file would be written to the subproject's
build directory and then would not be installed properly.
* Examples are no longer built on all travis images
* Coverage is no longer collected from all travis images
* Valgrind is no longer used with all travis images
This should greatly reduce the amount of compiling, downloading
binaries and general work the common images do.
This allows reducing the amount of friends needed for its interface
and some extra tricks later.
The bad part is that the pointer can become invalidated via
calls to other StringRef's public methods, but c'est la vie.
To prevent bugs with stitching system headers inside Catch,
the proxy header is responsible for guarding against inclusion
on Linux, rather than the includers.
Might be related to #1197
Create a namespaced Catch2::Catch target that is 'linkable' through
`target_link_libraries()` and export it so it is findable through
`find_package()`.
`find_package()` will find versions with the same major number and with
minor number >= requested.
This makes catch a lot easier to use in CMake-based projects. Whether it
is found using `find_package` or included in the client project as a
subdirectory, the client can include the catch headers per-target with
`target_include_directories(target PRIVATE Catch2::Catch).
Example usage:
cmake_minimum_required(VERSION 3.1)
# include Catch2 as subdirectory or installed package
# add_subdirectory(Catch2)
find_package(Catch2 VERSION 2.1.0 REQUIRED)
add_executable(tests tests/catch_main.cpp)
target_link_libraries(tests PRIVATE Catch2::Catch)
Also had to add new project to redirect CTest output, add
separate batch scripts for AppVeyor because it doesn't handle
multi-line batch scripts in yaml properly, and other helper
scripts.
* Every Linux build tracks coverage when running Debug mode
* OS X not supported yet (Future WIP)
* Our own unit tests, non-default reporters and Clara are ignored
ReusableStringStream holds a std::ostringstream internally, but only exposes the ostream interface.
It caches a pool of ostringstreams in a vector which is currently global, but will be made thread-local.
Altogether this should enable both runtime and compile-time benefits. although more work is needed to realise the compile time opportunities.
- Added new compilers and OS X images
- Option to run SelfTest under Valgrind
- Merge "Debug" and "Release" configurations into one run
-- This saves apt setup and cmake download step per compiler, 60-90s
- Fix C++14 compilation under Clang 3.8 and up
to prevent inheritance of include directories that possibly lead to a clash.
A clash occurs when a folder is included, e.g. examples, that wants to use the single-include directory instead of the normal include directory as used by the SelfTest in the next higher level.
1. Use it to conditionally define CATCH_INTERNAL_CONFIG_COUNTER, not
CATCH_CONFIG_COUNTER, as __JETBRAINS_IDE__ is similar to
compiler-provided macros, not to user-provided ones.
2. Since __COUNTER__ will work starting with CLion 2017.3, use it
when possible (and hopefully remove this check altogether
at some point).
Also hides std::chrono, std::pair and std::chrono::* behind
new configuration macros, CATCH_CONFIG_ENABLE_*_STRINGMAKER
to avoid dragging in <utility>, <tuple> and <chrono> in common
path, unless requested.
This works around a bug in libcxxrt handling of active exception count
that caused std::uncaught_exception() to return true even if there was
none.
Closes#1028
An empty default implementation is provided to keep backward compatibility.
Called when signal or Structured Exception is encountered.
Related to #1005
1. Use it to conditionally define CATCH_INTERNAL_CONFIG_COUNTER, not
CATCH_CONFIG_COUNTER, as __JETBRAINS_IDE__ is similar to
compiler-provided macros, not to user-provided ones.
2. Since __COUNTER__ will work starting with CLion 2017.3, use it
when possible (and hopefully remove this check altogether
at some point).
Swept:
`-Wpadded` in some places (where it caused extra size, instead of just
saying "hey, we padded struct at the end to align, just as standard says")
`-Wweak-vtables` everywhere (Clang)
`-Wexit-time-destructors` everywhere (Clang)
`-Wmissing-noreturn` everywhere (Clang)
The last three are enabled for Clang compilation going forward.
Also enabled `-Wunreachable-code` for Clang and GCC
A) non-const comparison operators should not exist and should not be
encouraged
B) The logic breaks comparing function pointers certain way
C) It was inconsistent anyway, as it only applied to `==` and `!=`
Closes#925
Note, this doesn't mean it will start passing, just that it will
run the approval tests properly
Some changes are needed before it passes, as the Windows output
somewhat differs.
* Assertions are defined into (void)(0) no-op
* SECTIONs are defined away (leaving {} as scope)
* TEST_CASEs and TEST_CASE_METHODs are not registered.
* REGISTER_TEST_CASE is defined into (void)(0) no-op
* METHOD_AS_TEST_CASE is defined away
-d takes "yes" or "no", rather than being a standalone flag option. The default is "defaultForReporter" so the previous change actually dropped some control.
This also introduces Catch::clog() method to allow embedded targets
to override std::clog usage with their own stream (presumably null-sink),
similarly to how Catch::cout() and Catch::cerr() are used.
Fixes#989
- this file excluded from the CATCH_CONFIG_DISABLE_MATCHERS path.
- matchers are always compiled in to the impl file
- _THROWS_WITH macros are still available with matchers disabled - but only the ones that take a string
- tests that use matchers have #ifdefs, so the whole SelfTest project can compile with matchers disable.
This should stop the evaluate machinery from instantiating all the
templates for every ptr type that is compared, instead generating it
for single one.
The toggle is `CATCH_CONFIG_DISABLE_MATCHERS` and the only use is
to speed up compilation of small TUs. For large ones it is likely
insignificant, because the speed up is constant relative to
number of tests/assertions in TU.
This should probably be done at the source of this external file, but I couldn't find where that is. Perhaps it should be mentioned in the file header comment?
This is not thread safe, but I think that was already true of Catch.
The construction/destruction of the std::ostringstream is where the
vast majority of time is spent per assertion. A simple test of
100000000 CHECK()s is reduced from around 60s to 7.4s
By using char const * instead of std::string we avoid significant
copying per assertion. In a simple loop with 10000000 CHECKS on
my system, this reduces the run time from 9.8s to 6s.
This fixes result disposition being ContinueOnFailure |
ContinueOnFailure for CHECK_THAT (obviously an error) and Normal |
ContinueOnFailure for REQUIRE_THAT (less obviously an error, but worse,
as that signals to the pipeline that assertion failure should both abort
and continue the test with ???? happening).
The whole GCC kinda sucks around warnings, this is yet another place
where pragmas manipulating warnings don't work properly and thus a
warning has to be disabled globally... luckily, this time it is
happening in selftest file and thus it isn't too problematic to just
turn that warning of for the entire file.
Jenkins groups junit test results by loosely interpreting the
classname attribute of the <testcase> element as a package-qualified
java class name such as java.util.String. It ignores the <testsuite>
elements in the xml. To organize test results we therefore need to
embed the suite name in the classname attribute as if it was a java
package name.
Fixes#922.
If [#filename] is present in tags, use it for the classname attribute,
rather than "global". If the test fixture is present that still takes
precedence.
Previously, some errors in Catch configuration would cause exceptions to
be thrown before main was even entered. This leads to call to
`std::terminate`, which is not a particularly nice way of ending the
binary.
Now these exceptions are registered with a global collector and used
once Catch enters main. They can also be optionally ignored, if user
supplies his own main and opts not to check them (or ignored them
intentionally).
Closes#921
This prevents Catch from disabling `Wpadded` for Clang inside test files
(files that do not define either `CATCH_CONFIG_MAIN` or
`CATCH_CONFIG_RUNNER`).
catch_suppress_warnings.h and catch_reenable_warnings.h should be
included only once*, so that the stitching script includes them as the
first and last header respectively, since it only includes each header
once. This caused a bug, where the first one was included properly, but
the second one was included prematurely, from catch_xmlwriter.hpp, and
thus was guarded by `CATCH_IMPL`.
* At least until the stitching script is changed to accomodate common
warning disabling header.
Fixes#871
Change it so the classname attribute on the <testcase> element is the
test fixture name or "global" regardless of whether the TEST_CASE
contains SECTIONs. This way the output is not changed substantially,
just because a SECTION is added to a TEST_CASE.
Types which are truthy, but have more information than the truthiness in their string conversion were showing up as 'true' or 'false' instead of showing the underlying type's string value.
If source files are defined using relative paths, CMake will compile the tests, but this script will (sometimes) fail to find and parse the tests from the source files. I have added an explicit warning when ParseAndAddCatchTests fails to find a source file.
Previously, this would not print out any messages for the last CHECK
```cpp
TEST_CASE("Foo") {
INFO("Test case start");
for (int i = 0; i < 2; ++i) {
INFO("The number is " << i);
CHECK(i == 0);
}
CHECK(false);
}
```
now it does.
Defining NO_SELFTEST=1 when cmake configuration is being done now turns
off SelfTest and Benchmark executables. This is for projects that
consume Catch using ExternalProject_Add and don't want to build our
selftest binaries for their unit test suite.
Closes#897
This means that `REQUIRE(std::vector<int>{1, 2} == std::vector<int>{1,
2});` works as expected.
Note that assertion macros taking more than 1 argument are currently not
variadic, because variadic args have to come last, which would make the
interface of these ugly: `REQUIRE_THROWS_AS(std::exception const&, ....
)`
All C++11 toggles are now removed. What is left is either platform
specific (POSIX_SIGNALS, WINDOWS_SEH), or possibly still needed
(USE_COUNTER).
If current CLion is compatible with `__COUNTER__`, then we should also
force `__COUNTER__` usage.
Changed
* CATCH_AUTO_PTR -> std::unique_ptr
* CATCH_OVERRIDE -> override
* CATCH_NULL -> nullptr
* CATCH_NOEXCEPT -> noexcept
* CATCH_NOEXCEPT_IS -> noexcept
Removed
* CATCH_CONFIG_CPP11_UNIQUE_PTR
* CATCH_CONFIG_CPP11_SHUFFLE
* CATCH_CONFIG_CPP11_TYPE_TRAITS
* CATCH_CONFIG_CPP11_OVERRIDE
* CATCH_CONFIG_CPP11_LONG_LONG
* CATCH_CONFIG_CPP11_TUPLE
* CATCH_CONFIG_CPP11_IS_ENUM
* CATCH_CONFIG_CPP11_GENERATED_METHODS
* CATCH_CONFIG_CPP11_NOEXCEPT
* CATCH_CONFIG_CPP11_NULLPTR
* CATCH_CONFIG_VARIADIC_MACROS
Added cmake script to parse the source files containing Catch's test and generate ctest definitions.
It generates one ctest test per `TEST_CASE` and labels them, using the `TEST_CASE`'s tags.
Closes#719.
- typedefs long long for MSVC
- typedefs uint64_t otherwise
Should probably do finer grained compiler checking - but this should at least be better than what was there before
When using C++11, comparison operators are already templated to take
anything that can be explicitly converted to double, but constructor
took only doubles. This lead to warnings when an `Approx` was
constructed from floats, which was problematic for some users.
Since just adding float constructor would be a large breaking change, as
suddenly `Approx( 1 )` would become ambiguous, I added a templated
constructor that will take anything that is explicitly convertible to
double. This has the added benefit of allowing constructing `Approx`
instances from instances of strong typedefs, ie allowing
`calculated_temp == Approx( known_temp)`.
Closes#873
Unexpected exceptions no longer cause abort and there should be no more
potential for false negatives.
The trade-off now is that exceptions are no longer translated.
This is another warning that follows test macros, making it painful to
suppress without leaking outside. Luckily clang's `_Pragma`
implementation works.
Should fix#308
Effectively a revert of previous commit, fixing #542, where this was
added to stop linters complaining about `REQUIRE_THROWS_AS` used like
`REQUIRE_THROWS_AS(expr, std::exception);`, which would be slicing the
caught exception. Now it is user's responsibility to pass us proper
exception type.
Closes#833 which wanted to add `typename`, so that the construct works
in a template, but that would not work with MSVC and older GCC's, as
having `typename` outside of a template is allowed only from C++11
onward.
This seems to give about 15% speedup when compiling tests using GCC.
The tradeoff is that under certain circumstances, there is a chance for
false negative result, when the expression under test throws exception
and the test code catches it before it gets to the test runner.
Example:
``` cpp
TEST_CASE("False negative") {
try {
REQUIRE(throws() == "");
} catch (...) {}
}
```
This test case will succeed, reporting no assertions checked, instead of
failing as it would with `CATCH_CONFIG_FAST_COMPILE` disabled. However,
just removing the try-catch block inside client's code will fix this, so
it is worthwhile.
This change does not apply to CHECK* macros, because these are currently
specified as continuing on exception and thus need the local try-catch
to work as intended.
std::ifstream in libstdc++ contains a bug, where it sets errno to zero.
To work around it, we manually save the errno before using std::ifstream
in debugger check, and reset it after we are done.
We also preventively save errno before using sprintf.
Fixes#835
VS 2015 in Release mode sees through our indirection and complains.
There is no reason to make the indirectoin harder to reason about,
instead of just disabling the warning.
In some cases, like when given
```cpp
std::vector<char>* str =
reinterpret_cast<std::vector<char>*>(0x1234458);
CHECK(*str == std::vector<char>());
```
reconstructing the expression to report it would cause another fatal
error. Instead we just put together an AssertionResult without
reconstructing the expression fully.
This should fully fix#810
If the gcc version supports `_Pragma` properly, we use that to disable
it locally inside assertions.
Otherwise we disable it for the entire TU.
Fixes#674
* Removed ccache (it was slowing down the compilation for some reason)
* Enabled some C++11 builds (gcc 4.8, gcc 6, clang 3.8 for now)
* Added gcc 4.4, 4.7 and clang 3.4
Some versions of MinGW do not support enough of Win32 API to let us work
with SEH, so SEH is now MSVC only (+ configurable toggle).
Also made use of gmtime_s MSVC only (as oposed to Windows only).
Fixes#805
It contains known limitation such as the fact that Catch is not thread
safe, it does not support running tests in forked process or running
multiple tests in parallel
closes#399closes#681closes#246closes#483
Because the signal changes were in a different branch from the windows.h
related changes, the SEH handling code included the header directly.
Fixes#803
The cmake download was failing, so we were relying on the old cmake,
which I broke recently (in 7ae96c710b).
Now the download should work again.
Also fixed warning that the requested OSX image no longer exists and
that it is automatically substituted for xcode 7.3 image.
This prevents Clang from complaining about unused value in expressions
containing explicit casts used in the THROW assertion macro family.
Example:
`REQUIRE_THROWS_AS(static_cast<bool>(object), std::bad_cast);` would
trigger `-Wunused-value` warning. Now it does not.
Credits to Arto Bendiken, who submitted a PR almost 3 years ago, but his
branch has since died and I was unable to merge it.
The integrated assembler segment was missing an underscore:
"_asm__". Also we remove the "DEBUG" macro check, so we are consistent
with the linux and windows variant.
Now breaking into gdb on failure should work via:
gdb --args test_executable --break
Mentioned that decomposing `&&` and `||` is not supported, gave examples
+ possible workarounds.
Closes#621, #787, #341 and maybe others I haven't found.
Using sizeof(expr) can trigger a compile-time error,
"lambda expressions are not allowed in an unevaluated context", when passing
expression containing lambda, like a std algorithm. This error is considered
a standard defect, as it is meant to prevent lambdas in decltype
or templates, but not in sizeof.
This reverts commit 227598af47.
In reality, this is a relatively small performance improvement,
especially with the previous improvements removing lots of superfluous
string handling, but still was measurable.
This is sane, because those `const char*`s are given to us by compiler,
from the text area and thus we do not have to manage their lifetimes. We
also never want to change them.
Also moved copy constructor to compiler-generated methods, not sure why
it wasn't -- even before it was the same as a compiler would generate.
This means that all tabs used in indentation are now 4 spaces and that
there should be no more trailing whitespace.
Ill also look into creating a pre-commit hook that will prevent this
from happening in the future.
Fixes#105
Rewrote main wrapping loop. Now uses iterators instead of indices and intermediate strings.
Differentiates between chars to wrap before, after or instead of.
Doesn’t preserve trailing newlines.
Wraps or more characters.
Dropped support for using tab character as an indent setting control char.
Hopefully avoids all the undefined behaviour and other bugs of the previous implementation.
Don't duplicate Catch::isDebuggerActive() check many times, do it just once
in CATCH_BREAK_INTO_DEBUGGER() definition and use a separate CATCH_TRAP()
macro for the really platform-dependent part.
* Empty strings are now direct constructed as `std::string()`, not as empty string literals.
* `startsWith` and `endsWith` no longer construct new a string. This should be an improvement
for libstdc++ when using older standards, as it doesn't use SSO but COW and thus even short
strings are expensive to first create.
* Various places now use char literal instead of string literals containing single char.
** `startsWith` and `endsWith` now also have overload that takes single char.
Generally the performance improvements under VS2015 are small, as going from short string
to char is mostly meaningless because of SSO (Catch doesn't push string handling that hard)
and previous commit removed most string handling if tests pass, which is the expect case.
This fixes the case when we pass signal to previously registered
handler, and it needs to transform the signal into different one.
Still problematic: What if the signal handler we replaced does not
terminate the application? We can end up in a weird state and loop
forever.
Possible solution: Deregister our signal handlers, CALL the previous
signal handler explicitly and if control returns, abort. This would
however complicate our code quite a bit, as we would have to parse the
sigaction we delegate to, decide whether to use signal handler or signal
action, etc...
Only some "signals" are handled under Windows, because Windows does not
use signals per-se and the mechanics are different. For now, we handle
sigsegv, stack overflow, div-by-zero and sigill. We can also
meaningfully
add various floating point errors, but not sigterm and family, because
sigterm is not a structured exception under Windows.
There is also no catch-all, because that would also catch various
debugger-related exceptions, like EXCEPTION_BREAKPOINT.
Also stops Catch from assuming its the only signal user in the binary,
and makes it restore the signal handlers it has replaced. Same goes for
the signal stack.
The signal stack itself probably shouldn't be always reallocated for
fragmentation reasons, but that can be fixed later on.
Now if we detect C++11 compiler, or MSVC in version corresponding to VS2015,
we switch from using `std::random_shuffle` to `std::shuffle`.
`std::random_shuffle` was officially deprecated in C++14, and removed in C++17.
Also removed guarded inclusion of `<random>` header, as there was nothing
in the header that used it.
Before it was taken from whatever last build happened, which led it show
a build error because I took PR against wrong branch.
This should be fixed now.
It's = it is
In the sentence "In fact it is usually a good idea to put the block with the ```#define``` [in it's own source file](slow-compiles.md).", the correct usage is "its", not "it's".
Catch passes ::tolower into std::transform with string iterators.
::tolower has the signature int(int), which triggers a stealth narrowing
warning inside std::transform, because transform calls
*_Dest = _Fn(*_First), which implicitly narrows an int to a char.
For this particular application the narrowing is fine, so explicitly
narrow in a lambda.
Catch passes an RNG which accepts int to random_shuffle. Inside
random_shuffle, the STL tries to call that RNG with the difference_type
of the user provided iterators. For std::vector, this is ptrdiff_t,
which on amd64 builds is wider than int. This triggers a narrowing
warning because the 64 bit difference is being truncated to 32 bits.
Note that this RNG implementation still does not produce a correctly
uniformly shuffled result -- it's currently asserting that std::rand
can produce 1000000 which is false -- but I don't know enough about
how much repeatable shuffles are necessary here, so I'm leaving that
alone for now.
SCENARIO does not add leading spaces to the test name (only BDD-style section
names are modified), so the trimming is not necessary. But if the name is
trimmed, it makes it harder to correlate the output of XML reporter with tests
that have leading spaces in their name: e.g. these tests will have the same name
attribute:
TEST_CASE("Test") {}
TEST_CASE(" Test") {}
This commit fixes the following scenario:
* You have a test that compares strings with embedded control
characters.
* The test fails.
* You are using JUnit tests within TeamCity.
Before this commit, the JUnit report watcher fails on parsing the XML
for two reasons: the control characters are missing a semicolon at the
end, and the XML document doesn't specify that it is XML 1.1.
XML 1.0 --- what we get if we don't specify an XML version --- doesn't support embedding control characters --- see
http://stackoverflow.com/questions/404107/why-are-control-characters-illegal-in-xml
for all of the gory details.
This is based on PR #588 by @mrpi
Instead of `exit(1)`, it now throws `std::runtime_error` with the details
of the failure. This exception is handled in `run()` at a higher level where
the log is printed to cerr and the test gracefully exits.
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at github@philnash.me. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
[](https://discord.gg/4CWS9zD)
Build status (on Travis CI) [](https://travis-ci.org/philsquared/Catch)
<ahref="https://raw.githubusercontent.com/philsquared/Catch/master/single_include/catch.hpp">The latest, single header, version can be downloaded directly using this link</a>
## Catch2 is released!
If you've been using an earlier version of Catch, please see the
Breaking Changes section of [the release notes](https://github.com/catchorg/Catch2/releases/tag/v2.0.1)
before moving to Catch2. You might also like to read [this blog post](https://levelofindirection.com/blog/catch2-released.html) for more details.
## What's the Catch?
Catch stands for C++ Automated Test Cases in Headers and is a multi-paradigm automated test framework for C++ and Objective-C (and, maybe, C). It is implemented entirely in a set of header files, but is packaged up as a single header for extra convenience.
Catch2 is a multi-paradigm test framework for C++.
## How to use it
This documentation comprises these three parts:
* [Why do we need yet another C++ Test Framework?](docs/why-catch.md)
* [Tutorial](docs/tutorial.md) - getting started
* [Reference section](docs/Readme.md) - all the details
* [Why do we need yet another C++ Test Framework?](docs/why-catch.md#top)
* [Tutorial](docs/tutorial.md#top) - getting started
* [Reference section](docs/Readme.md#top) - all the details
## More
* Issues and bugs can be raised on the [Issue tracker on GitHub](https://github.com/philsquared/Catch/issues)
* For discussion or questions please use [the dedicated Google Groups forum](https://groups.google.com/forum/?fromgroups#!forum/catch-forum)
* Issues and bugs can be raised on the [Issue tracker on GitHub](https://github.com/catchorg/Catch2/issues)
* For discussion or questions please use [the dedicated Google Groups forum](https://groups.google.com/forum/?fromgroups#!forum/catch-forum) or our [Discord](https://discord.gg/4CWS9zD)
* See [who else is using Catch2](docs/opensource-users.md#top)
[Expressions with commas](#expressions-with-commas)<br>
Most test frameworks have a large collection of assertion macros to capture all possible conditional forms (```_EQUALS```, ```_NOTEQUALS```, ```_GREATER_THAN``` etc).
Catch is different. Because it decomposes natural C-style conditional expressions most of these forms are reduced to one or two that you will use all the time. That said there are a rich set of auxilliary macros as well. We'll describe all of these here.
Catch is different. Because it decomposes natural C-style conditional expressions most of these forms are reduced to one or two that you will use all the time. That said there is a rich set of auxiliary macros as well. We'll describe all of these here.
Most of these macros come in two forms:
@@ -14,7 +22,7 @@ The ```CHECK``` family are equivalent but execution continues in the same test c
* **REQUIRE(** _expression_ **)** and
* **CHECK(** _expression_ **)**
Evaluates the expression and records the result. If an exception is thrown it is caught, reported, and counted as a failure. These are the macros you will use most of the time
Evaluates the expression and records the result. If an exception is thrown, it is caught, reported, and counted as a failure. These are the macros you will use most of the time.
Examples:
```
@@ -34,26 +42,76 @@ Example:
REQUIRE_FALSE( thisReturnsFalse() );
```
Do note that "overly complex" expressions cannot be decomposed and thus will not compile. This is done partly for practical reasons (to keep the underlying expression template machinery to minimum) and partly for philosophical reasons (assertions should be simple and deterministic).
Examples:
* `CHECK(a == 1 && b == 2);`
This expression is too complex because of the `&&` operator. If you want to check that 2 or more properties hold, you can either put the expression into parenthesis, which stops decomposition from working, or you need to decompose the expression into two assertions: `CHECK( a == 1 ); CHECK( b == 2);`
* `CHECK( a == 2 || b == 1 );`
This expression is too complex because of the `||` operator. If you want to check that one of several properties hold, you can put the expression into parenthesis (unlike with `&&`, expression decomposition into several `CHECK`s is not possible).
### Floating point comparisons
When comparing floating point numbers - especially if at least one of them has been computed - great care must be taken to allow for rounding errors and inexact representations.
Catch provides a way to perform tolerant comparisons of floating point values through use of a wrapper class called ```Approx```. ```Approx``` can be used on either side of a comparison expression. It overloads the comparisons operators to take a tolerance into account. Here's a simple example:
Catch provides a way to perform tolerant comparisons of floating point values through use of a wrapper class called `Approx`. `Approx` can be used on either side of a comparison expression. It overloads the comparisons operators to take a tolerance into account. Here's a simple example:
```
```cpp
REQUIRE( performComputation() == Approx( 2.1 ) );
```
By default a small epsilon value is used that covers many simple cases of rounding errors. When this is insufficent the epsilon value (the amount within which a difference either way is ignored) can be specified by calling the ```epsilon()``` method on the ```Approx``` instance. e.g.:
Catch also provides a user-defined literal for `Approx`; `_a`. It resides in
the `Catch::literals` namespace and can be used like so:
```cpp
using namespace Catch::literals;
REQUIRE( performComputation() == 2.1_a );
```
When dealing with very large or very small numbers it can be useful to specify a scale, which can be achieved by calling the ```scale()``` method on the ```Approx``` instance.
`Approx` is constructed with defaults that should cover most simple cases.
For the more complex cases, `Approx` provides 3 customization points:
* __epsilon__ - epsilon serves to set the coefficient by which a result
can differ from `Approx`'s value before it is rejected.
_By default set to `std::numeric_limits<float>::epsilon()*100`._
* __margin__ - margin serves to set the the absolute value by which
a result can differ from `Approx`'s value before it is rejected.
_By default set to `0.0`._
* __scale__ - scale is used to change the magnitude of `Approx` for relative check.
_By default set to `0.0`._
#### epsilon example
```cpp
Approx target = Approx(100).epsilon(0.01);
100.0 == target; // Obviously true
200.0 == target; // Obviously still false
100.5 == target; // True, because we set target to allow up to 1% difference
```
#### margin example
```cpp
Approx target = Approx(100).margin(5);
100.0 == target; // Obviously true
200.0 == target; // Obviously still false
104.0 == target; // True, because we set target to allow absolute difference of at most 5
```
#### scale
Scale can be useful if the computation leading to the result worked
on different scale than is used by the results. Since allowed difference
between Approx's value and compared value is based primarily on Approx's value
(the allowed difference is computed as
`(Approx::scale + Approx::value) * epsilon`), the resulting comparison could
need rescaling to be correct.
## Exceptions
* **REQUIRE_NOTHROW(** _expression_ **)** and
* **CHECK_NOTHROW(** _expression_ **)**
Expects that no exception is thrown during evaluation of the expression.
* **REQUIRE_THROWS(** _expression_ **)** and
* **CHECK_THROWS(** _expression_ **)**
@@ -62,21 +120,82 @@ Expects that an exception (of any type) is be thrown during evaluation of the ex
* **REQUIRE_THROWS_AS(** _expression_, _exception type_ **)** and
Expects that an exception of the _specified type_ is thrown during evaluation of the expression.
Expects that an exception of the _specified type_ is thrown during evaluation of the expression. Note that the _exception type_ is extended with `const&` and you should not include it yourself.
* **REQUIRE_THROWS_WITH(** _expression_, _string or string matcher_ **)** and
* **CHECK_THROWS_WITH(** _expression_, _string or string matcher_ **)**
Expects that an exception is thrown that, when converted to a string, matches the _string_ or _string matcher_ provided (see next section for Matchers).
REQUIRE_THROWS_WITH( dismantleHal(), "My mind is going" );
```
* **REQUIRE_THROWS_MATCHES(** _expression_, _exception type_, _matcher for given exception type_ **)** and
* **CHECK_THROWS_MATCHES(** _expression_, _exception type_, _matcher for given exception type_ **)**
Expects that exception of _exception type_ is thrown and it matches provided matcher (see the [documentation for Matchers](matchers.md#top)).
_Please note that the `THROW` family of assertions expects to be passed a single expression, not a statement or series of statements. If you want to check a more complicated sequence of operations, you can use a C++11 lambda function._
```cpp
REQUIRE_NOTHROW([&](){
int i = 1;
int j = 2;
auto k = i + j;
if (k == 3) {
throw 1;
}
}());
```
* **REQUIRE_NOTHROW(** _expression_**)** and
* **CHECK_NOTHROW(** _expression_**)**
Expects that no exception is thrown during evaluation of the expression.
## Matcher expressions
To support Matchers a slightly different form is used. Matchers will be more fully documented elsewhere. *Note that Matchers are still at early stage development and are subject to change.*
To support Matchers a slightly different form is used. Matchers have [their own documentation](matchers.md#top).
* **REQUIRE_THAT(** _lhs_, _matcher call_ **)** and
* **CHECK_THAT(** _lhs_, _matcher call_ **)**
* **REQUIRE_THAT(** _lhs_, _matcher expression_ **)** and
Build Systems may refer to low-level tools, like CMake, or larger systems that run on servers, like Jenkins or TeamCity. This page will talk about both.
# Continuous Integration systems
Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (as has been done with TeamCity).
Two of these reporters are built in (XML and JUnit) and the third (TeamCity) is included as a separate header. It's possible that the other two may be split out in the future too - as that would make the core of Catch smaller for those that don't need them.
## XML Reporter
```-r xml```
The XML Reporter writes in an XML format that is specific to Catch.
The advantage of this format is that it corresponds well to the way Catch works (especially the more unusual features, such as nested sections) and is a fully streaming format - that is it writes output as it goes, without having to store up all its results before it can start writing.
The disadvantage is that, being specific to Catch, no existing build servers understand the format natively. It can be used as input to an XSLT transformation that could covert it to, say, HTML - although this loses the streaming advantage, of course.
## JUnit Reporter
```-r junit```
The JUnit Reporter writes in an XML format that mimics the JUnit ANT schema.
The advantage of this format is that the JUnit Ant schema is widely understood by most build servers and so can usually be consumed with no additional work.
The disadvantage is that this schema was designed to correspond to how JUnit works - and there is a significant mismatch with how Catch works. Additionally the format is not streamable (because opening elements hold counts of failed and passing tests as attributes) - so the whole test run must complete before it can be written.
## TeamCity Reporter
```-r teamcity```
The TeamCity Reporter writes TeamCity service messages to stdout. In order to be able to use this reporter an additional header must also be included.
```catch_reporter_teamcity.hpp``` can be found in the ```include\reporters``` directory. It should be included in the same file that ```#define```s ```CATCH_CONFIG_MAIN``` or ```CATCH_CONFIG_RUNNER```. The ```#include``` should be placed after ```#include```ing Catch itself.
e.g.:
```
#define CATCH_CONFIG_MAIN
#include "catch.hpp"
#include "catch_reporter_teamcity.hpp"
```
Being specific to TeamCity this is the best reporter to use with it - but it is completely unsuitable for any other purpose. It is a streaming format (it writes as it goes) - although test results don't appear in the TeamCity interface until the completion of a suite (usually the whole test run).
# Low-level tools
## CMake
You can use the following CMake script to automatically fetch Catch from github and configure it as an external project:
# Expose required variable (CATCH_INCLUDE_DIR) to parent scope
ExternalProject_Get_Property(catch source_dir)
set(CATCH_INCLUDE_DIR ${source_dir}/include CACHE INTERNAL "Path to include folder for Catch")
```
If you put it in, e.g., `${PROJECT_SRC_DIR}/${EXT_PROJECTS_DIR}/catch/`, you can use it in your project by adding the following to your root CMake file:
This page talks about how Catch integrates with Continuous Integration
Build Systems may refer to low-level tools, like CMake, or larger systems that run on servers, like Jenkins or TeamCity. This page will talk about both.
## Continuous Integration systems
Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (currently we also offer TeamCity, TAP, Automake and SonarQube reporters).
Two of these reporters are built in (XML and JUnit) and the third (TeamCity) is included as a separate header. It's possible that the other two may be split out in the future too - as that would make the core of Catch smaller for those that don't need them.
### XML Reporter
```-r xml```
The XML Reporter writes in an XML format that is specific to Catch.
The advantage of this format is that it corresponds well to the way Catch works (especially the more unusual features, such as nested sections) and is a fully streaming format - that is it writes output as it goes, without having to store up all its results before it can start writing.
The disadvantage is that, being specific to Catch, no existing build servers understand the format natively. It can be used as input to an XSLT transformation that could convert it to, say, HTML - although this loses the streaming advantage, of course.
### JUnit Reporter
```-r junit```
The JUnit Reporter writes in an XML format that mimics the JUnit ANT schema.
The advantage of this format is that the JUnit Ant schema is widely understood by most build servers and so can usually be consumed with no additional work.
The disadvantage is that this schema was designed to correspond to how JUnit works - and there is a significant mismatch with how Catch works. Additionally the format is not streamable (because opening elements hold counts of failed and passing tests as attributes) - so the whole test run must complete before it can be written.
## Other reporters
Other reporters are not part of the single-header distribution and need
to be downloaded and included separately. All reporters are stored in
`single_include` directory in the git repository, and are named
`catch_reporter_*.hpp`. For example, to use the TeamCity reporter you
need to download `single_include/catch_reporter_teamcity.hpp` and include
it after Catch itself.
```cpp
#define CATCH_CONFIG_MAIN
#include "catch.hpp"
#include "catch_reporter_teamcity.hpp"
```
### TeamCity Reporter
```-r teamcity```
The TeamCity Reporter writes TeamCity service messages to stdout. In order to be able to use this reporter an additional header must also be included.
Being specific to TeamCity this is the best reporter to use with it - but it is completely unsuitable for any other purpose. It is a streaming format (it writes as it goes) - although test results don't appear in the TeamCity interface until the completion of a suite (usually the whole test run).
### Automake Reporter
```-r automake```
The Automake Reporter writes out the [meta tags](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html#Log-files-generation-and-test-results-recording) expected by automake via `make check`.
### TAP (Test Anything Protocol) Reporter
```-r tap```
Because of the incremental nature of Catch's test suites and ability to run specific tests, our implementation of TAP reporter writes out the number of tests in a suite last.
### SonarQube Reporter
```-r sonarqube```
[SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format for tests metrics.
## Low-level tools
### Precompiled headers (PCHs)
Catch offers prototypal support for being included in precompiled headers, but because of its single-header nature it does need some actions by the user:
* The precompiled header needs to define `CATCH_CONFIG_ALL_PARTS`
* define `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`
* include "catch.hpp" again
### CodeCoverage module (GCOV, LCOV...)
If you are using GCOV tool to get testing coverage of your code, and are not sure how to integrate it with CMake and Catch, there should be an external example over at https://github.com/fkromer/catch_cmake_coverage
### pkg-config
Catch2 provides a rudimentary pkg-config integration, by registering itself
under the name `catch2`. This means that after Catch2 is installed, you
can use `pkg-config` to get its include path: `pkg-config --cflags catch2`.
### gdb and lldb scripts
Catch2's `contrib` folder also contains two simple debugger scripts,
`gdbinit` for `gdb` and `lldbinit` for `lldb`. If loaded into their
respective debugger, these will tell it to step over Catch2's internals
when stepping through code.
## CMake
[As it has been getting kinda long, the documentation of Catch2's
integration with CMake has been moved to its own page.](cmake-integration.md#top)
[Load test names to run from a file](#load-test-names-to-run-from-a-file)<br>
[Just test names](#just-test-names)<br>
[Specify the order test cases are run](#specify-the-order-test-cases-are-run)<br>
[Specify a seed for the Random Number Generator](#specify-a-seed-for-the-random-number-generator)<br>
[Identify framework and version according to the libIdentify standard](#identify-framework-and-version-according-to-the-libidentify-standard)<br>
[Wait for key before continuing](#wait-for-key-before-continuing)<br>
[Specify the number of benchmark samples to collect](#specify-the-number-of-benchmark-samples-to-collect)<br>
[Specify the number of resamples for bootstrapping](#specify-the-number-of-resamples-for-bootstrapping)<br>
[Specify the confidence-interval for bootstrapping](#specify-the-confidence-interval-for-bootstrapping)<br>
[Disable statistical analysis of collected benchmark samples](#disable-statistical-analysis-of-collected-benchmark-samples)<br>
[Specify the amount of time in milliseconds spent on warming up each test](#specify-the-amount-of-time-in-milliseconds-spent-on-warming-up-each-test)<br>
[Usage](#usage)<br>
[Specify the section to run](#specify-the-section-to-run)<br>
Catch works quite nicely without any command line options at all - but for those times when you want greater control the following options are available.
Click one of the followings links to take you straight to that option - or scroll on to browse the available options.
Click one of the following links to take you straight to that option - or scroll on to browse the available options.
@@ -45,7 +88,7 @@ Wildcards consist of the `*` character at the beginning and/or end of test case
Test specs are case insensitive.
If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precendence, however.
If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precedence, however.
Inclusions and exclusions are evaluated in left-to-right order.
Test case examples:
@@ -58,14 +101,17 @@ exclude:notThis Matches all tests except, 'notThis'
~*private* Matches all tests except those that contain 'private'
a* ~ab* abc Matches all tests that start with 'a', except those that
start with 'ab', except 'abc', which is included
-# [#somefile] Matches all tests from the file 'somefile.cpp'
</pre>
Names within square brackets are interpreted as tags.
A series of tags form an AND expression wheras a comma-separated sequence forms an OR expression. e.g.:
A series of tags form an AND expression whereas a comma-separated sequence forms an OR expression. e.g.:
<pre>[one][two],[three]</pre>
This matches all tests tagged `[one]` and `[two]`, as well as all tests tagged `[three]`
Test names containing special characters, such as `,` or `[` can specify them on the command line using `\`.
`\` also escapes itself.
<aid="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use
@@ -87,8 +133,9 @@ The JUnit reporter is an xml format that follows the structure of the JUnit XML
## Breaking into the debugger
<pre>-b, --break</pre>
In some IDEs (currently XCode and Visual Studio) it is possible for Catch to break into the debugger on a test failure. This can be very helpful during debug sessions - especially when there is more than one path through a particular test.
In addition to the command line option, ensure you have built your code with the DEBUG preprocessor symbol
Under most debuggers Catch2 is capable of automatically breaking on a test
failure. This allows the user to see the current state of the test during
failure.
<aid="showing-results-for-successful-tests"></a>
## Showing results for successful tests
@@ -115,7 +162,7 @@ Sometimes this results in a flood of failure messages and you'd rather just see
--list-reporters
</pre>
```-l``` or ```--list-tests`` will list all registered tests, along with any tags.
```-l``` or ```--list-tests``` will list all registered tests, along with any tags.
If one or more test-specs have been supplied too then only the matching tests will be listed.
```-t``` or ```--list-tags``` lists all available tags, along with the number of test cases they match. Again, supplying test specs limits the tags that match.
@@ -158,9 +205,16 @@ This option transforms tabs and newline characters into ```\t``` and ```\n``` re
## Warnings
<pre>-w, --warn <warning name></pre>
Enables reporting of warnings (only one, at time of this writing). If a warning is issued it fails the test.
Enables reporting of suspicious test states. There are currently two
available warnings
```
NoAssertions // Fail test case / leaf section if no assertions
// (e.g. `REQUIRE`) is encountered.
NoTests // Return non-zero exit code when no test cases were run
// Also calls reporter's noMatchingTestCases method
```
The ony available warning, presently, is ```NoAssertions```. This warning fails a test case, or (leaf) section if no assertions (```REQUIRE```/ ```CHECK``` etc) are encountered.
<a id="reporting-timings"></a>
## Reporting timings
@@ -189,15 +243,25 @@ This option lists all available tests in a non-indented form, one on each line.
Test cases are ordered one of three ways:
### decl
Declaration order. The order the tests were originally declared in. Note that ordering between files is not guaranteed and is implementation dependent.
Declaration order (this is the default order if no --order argument is provided).
Tests in the same TU are sorted using their declaration orders, different
TUs are in an implementation (linking) dependent order.
### lex
Lexicographically sorted. Tests are sorted, alpha-numerically, by name.
Lexicographic order. Tests are sorted by their name, their tags are ignored.
### rand
Randomly sorted. Test names are sorted using ```std::random_shuffle()```. By default the random number generator is seeded with 0 - and so the order is repeatable. To control the random seed see <a href="#rng-seed">rng-seed</a>.
Randomly sorted. The order is dependent on Catch2's random seed (see
[`--rng-seed`](#rng-seed)), and is subset invariant. What this means
is that as long as the random seed is fixed, running only some tests
(e.g. via tag) does not change their relative order.
> The subset stability was introduced in Catch2 v2.12.0
<a id="rng-seed"></a>
## Specify a seed for the Random Number Generator
@@ -205,16 +269,143 @@ Randomly sorted. Test names are sorted using ```std::random_shuffle()```. By def
Sets a seed for the random number generator using ```std::srand()```.
If a number is provided this is used directly as the seed so the random pattern is repeatable.
Alternatively if the keyword ```time``` is provided then the result of calling ```std::time(0)``` is used and so the pattern becomes unpredictable.
Alternatively if the keyword ```time``` is provided then the result of calling ```std::time(0)``` is used and so the pattern becomes unpredictable. In some cases, you might need to pass the keyword ```time``` in double quotes instead of single quotes.
In either case the actual value for the seed is printed as part of Catch's output so if an issue is discovered that is sensitive to test ordering the ordering can be reproduced - even if it was originally seeded from ```std::time(0)```.
<a id="libidentify"></a>
## Identify framework and version according to the libIdentify standard
<pre>--libidentify</pre>
See [The LibIdentify repo for more information and examples](https://github.com/janwilmans/LibIdentify).
Catch is designed to "just work" as much as possible. For most people the only configuration needed is telling Catch which source file should host all the implementation code (```CATCH_CONFIG_MAIN```).
Nonetheless there are still some occasions where finer control is needed. For these occasions Catch exposes a small set of macros for configuring how it is built.
Nonetheless there are still some occasions where finer control is needed. For these occasions Catch exposes a set of macros for configuring how it is built.
# main()/ implementation
CATCH_CONFIG_MAIN // Designates this as implementation file and defines main()
CATCH_CONFIG_RUNNER // Designates this as implementation file
## Prefixing Catch macros
Although Catch is header only it still, internally, maintains a distinction between interface headers and headers that contain implementation. Only one source file in your test project should compile the implementation headers and this is controlled through the use of one of these macros - one of these identifiers should be defined before including Catch in *exactly one implementation file in your project*.
# Prefixing Catch macros
CATCH_CONFIG_PREFIX_ALL
CATCH_CONFIG_PREFIX_ALL
To keep test code clean and uncluttered Catch uses short macro names (e.g. ```TEST_CASE``` and ```REQUIRE```). Occasionally these may conflict with identifiers from platform headers or the system under test. In this case the above identifier can be defined. This will cause all the Catch user macros to be prefixed with ```CATCH_``` (e.g. ```CATCH_TEST_CASE``` and ```CATCH_REQUIRE```).
# Terminal colour
## Terminal colour
CATCH_CONFIG_COLOUR_NONE// completely disables all text colouring
CATCH_CONFIG_COLOUR_WINDOWS// forces the Win32 console API to be used
CATCH_CONFIG_COLOUR_ANSI// forces ANSI colour codes to be used
CATCH_CONFIG_COLOUR_NONE// completely disables all text colouring
CATCH_CONFIG_COLOUR_WINDOWS// forces the Win32 console API to be used
CATCH_CONFIG_COLOUR_ANSI// forces ANSI colour codes to be used
Yes, I am English, so I will continue to spell "colour" with a 'u'.
@@ -32,42 +44,211 @@ Note that when ANSI colour codes are used "unistd.h" must be includable - along
Typically you should place the ```#define``` before #including "catch.hpp" in your main source file - but if you prefer you can define it for your whole project by whatever your IDE or build system provides for you to do so.
# Console width
## Console width
CATCH_CONFIG_CONSOLE_WIDTH = x // where x is a number
CATCH_CONFIG_CONSOLE_WIDTH = x // where x is a number
Catch formats output intended for the console to fit within a fixed number of characters. This is especially important as indentation is used extensively and uncontrolled line wraps break this.
By default a console width of 80 is assumed but this can be controlled by defining the above identifier to be a different value.
# stdout
## stdout
CATCH_CONFIG_NOSTDOUT
CATCH_CONFIG_NOSTDOUT
Catch does not use ```std::cout``` and ```std::cerr``` directly but gets them from ```Catch::cout()``` and ```Catch::cerr()``` respectively. If the above identifier is defined these functions are left unimplemented and you must implement them yourself. Their signatures are:
To support platforms that do not provide `std::cout`, `std::cerr` and
`std::clog`, Catch does not usem the directly, but rather calls
`Catch::cout`, `Catch::cerr` and `Catch::clog`. You can replace their
implementation by defining `CATCH_CONFIG_NOSTDOUT` and implementing
them yourself, their signatures are:
std::ostream& cout();
std::ostream& cerr();
std::ostream& clog();
This can be useful on certain platforms that do not provide ```std::cout``` and ```std::cerr```, such as certain embedded systems.
[You can see an example of replacing these functions here.](
../examples/231-Cfg-OutputStreams.cpp)
# C++ conformance toggles
CATCH_CONFIG_CPP11_NULLPTR // nullptr is supported?
CATCH_CONFIG_CPP11_NOEXCEPT // noexcept is supported?
CATCH_CONFIG_CPP11_GENERATED_METHODS // delete and default keywords for methods
CATCH_CONFIG_CPP11_IS_ENUM // std::is_enum is supported?
CATCH_CONFIG_CPP11_TUPLE // std::tuple is supported
CATCH_CONFIG_VARIADIC_MACROS // Usually pre-C++11 compiler extensions are sufficient
CATCH_CONFIG_CPP11_LONG_LONG // generates overloads for the long long type
CATCH_CONFIG_CPP11_OVERRIDE // CATCH_OVERRIDE expands to override (for virtual function implementations)
CATCH_CONFIG_CPP11_UNIQUE_PTR // Use std::unique_ptr instead of std::auto_ptr
## Fallback stringifier
Catch has some basic compiler detection that will attempt to select the appropriate mix of these macros. However being incomplete - and often without access to the respective compilers - this detection tends to be conservative.
So overriding control is given to the user. If a compiler supports a feature (and Catch does not already detect it) then one or more of these may be defined to enable it (or suppress it, in some cases). If you do do this please raise an issue, specifying your compiler version (ideally with an idea of how to detect it) and stating that it has such support.
You may also suppress any of these features by using the `_NO_` form, e.g. `CATCH_CONFIG_CPP11_NO_NULLPTR`.
By default, when Catch's stringification machinery has to stringify
a type that does not specialize `StringMaker`, does not overload `operator<<`,
is not an enumeration and is not a range, it uses `"{?}"`. This can be
overridden by defining `CATCH_CONFIG_FALLBACK_STRINGIFIER` to name of a
function that should perform the stringification instead.
All types that do not provide `StringMaker` specialization or `operator<<`
overload will be sent to this function (this includes enums and ranges).
The provided function must return `std::string` and must accept any type,
e.g. via overloading.
_Note that if the provided function does not handle a type and this type
requires to be stringified, the compilation will fail._
## Default reporter
Catch's default reporter can be changed by defining macro
`CATCH_CONFIG_DEFAULT_REPORTER` to string literal naming the desired
default reporter.
This means that defining `CATCH_CONFIG_DEFAULT_REPORTER` to `"console"`
is equivalent with the out-of-the-box experience.
## C++11 toggles
CATCH_CONFIG_CPP11_TO_STRING // Use `std::to_string`
Because we support platforms whose standard library does not contain
`std::to_string`, it is possible to force Catch to use a workaround
based on `std::stringstream`. On platforms other than Android,
the default is to use `std::to_string`. On Android, the default is to
use the `stringstream` workaround. As always, it is possible to override
Catch's selection, by defining either `CATCH_CONFIG_CPP11_TO_STRING` or
`CATCH_CONFIG_NO_CPP11_TO_STRING`.
## C++17 toggles
CATCH_CONFIG_CPP17_UNCAUGHT_EXCEPTIONS // Override std::uncaught_exceptions (instead of std::uncaught_exception) support detection
CATCH_CONFIG_CPP17_STRING_VIEW // Override std::string_view support detection (Catch provides a StringMaker specialization by default)
CATCH_CONFIG_CPP17_VARIANT // Override std::variant support detection (checked by CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER)
CATCH_CONFIG_CPP17_OPTIONAL // Override std::optional support detection (checked by CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER)
CATCH_CONFIG_CPP17_BYTE // Override std::byte support detection (Catch provides a StringMaker specialization by default)
> `CATCH_CONFIG_CPP17_STRING_VIEW` was [introduced](https://github.com/catchorg/Catch2/issues/1376) in Catch 2.4.1.
Catch contains basic compiler/standard detection and attempts to use
some C++17 features whenever appropriate. This automatic detection
can be manually overridden in both directions, that is, a feature
can be enabled by defining the macro in the table above, and disabled
by using `_NO_` in the macro, e.g. `CATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS`.
## Other toggles
CATCH_CONFIG_COUNTER // Use __COUNTER__ to generate unique names for test cases
CATCH_CONFIG_WINDOWS_SEH // Enable SEH handling on Windows
CATCH_CONFIG_FAST_COMPILE // Sacrifices some (rather minor) features for compilation speed
CATCH_CONFIG_WINDOWS_CRTDBG // Enable leak checking using Windows's CRT Debug Heap
CATCH_CONFIG_DISABLE_STRINGIFICATION // Disable stringifying the original expression
CATCH_CONFIG_DISABLE // Disables assertions and test case registration
CATCH_CONFIG_WCHAR // Enables use of wchart_t
CATCH_CONFIG_EXPERIMENTAL_REDIRECT // Enables the new (experimental) way of capturing stdout/stderr
CATCH_CONFIG_USE_ASYNC // Force parallel statistical processing of samples during benchmarking
CATCH_CONFIG_ANDROID_LOGWRITE // Use android's logging system for debug output
CATCH_CONFIG_GLOBAL_NEXTAFTER // Use nextafter{,f,l} instead of std::nextafter
> [`CATCH_CONFIG_ANDROID_LOGWRITE`](https://github.com/catchorg/Catch2/issues/1743) and [`CATCH_CONFIG_GLOBAL_NEXTAFTER`](https://github.com/catchorg/Catch2/pull/1739) were introduced in Catch 2.10.0
Currently Catch enables `CATCH_CONFIG_WINDOWS_SEH` only when compiled with MSVC, because some versions of MinGW do not have the necessary Win32 API support.
`CATCH_CONFIG_POSIX_SIGNALS` is on by default, except when Catch is compiled under `Cygwin`, where it is disabled by default (but can be force-enabled by defining `CATCH_CONFIG_POSIX_SIGNALS`).
`CATCH_CONFIG_WINDOWS_CRTDBG` is off by default. If enabled, Windows's
CRT is used to check for memory leaks, and displays them after the tests
finish running. This option only works when linking against the default
main, and must be defined for the whole library build.
`CATCH_CONFIG_WCHAR` is on by default, but can be disabled. Currently
it is only used in support for DJGPP cross-compiler.
With the exception of `CATCH_CONFIG_EXPERIMENTAL_REDIRECT`,
these toggles can be disabled by using `_NO_` form of the toggle,
e.g. `CATCH_CONFIG_NO_WINDOWS_SEH`.
### `CATCH_CONFIG_FAST_COMPILE`
This compile-time flag speeds up compilation of assertion macros by ~20%,
by disabling the generation of assertion-local try-catch blocks for
non-exception family of assertion macros ({`REQUIRE`,`CHECK`}{``,`_FALSE`, `_THAT`}).
This disables translation of exceptions thrown under these assertions, but
should not lead to false negatives.
`CATCH_CONFIG_FAST_COMPILE` has to be either defined, or not defined,
in all translation units that are linked into single test binary.
### `CATCH_CONFIG_DISABLE_STRINGIFICATION`
This toggle enables a workaround for VS 2017 bug. For details see [known limitations](limitations.md#visual-studio-2017----raw-string-literal-in-assert-fails-to-compile).
### `CATCH_CONFIG_DISABLE`
This toggle removes most of Catch from given file. This means that `TEST_CASE`s are not registered and assertions are turned into no-ops. Useful for keeping tests within implementation files (ie for functions with internal linkage), instead of in external files.
This feature is considered experimental and might change at any point.
_Inspired by Doctest's `DOCTEST_CONFIG_DISABLE`_
## Windows header clutter
On Windows Catch includes `windows.h`. To minimize global namespace clutter in the implementation file, it defines `NOMINMAX` and `WIN32_LEAN_AND_MEAN` before including it. You can control this behaviour via two macros:
CATCH_CONFIG_NO_NOMINMAX // Stops Catch from using NOMINMAX macro
CATCH_CONFIG_NO_WIN32_LEAN_AND_MEAN // Stops Catch from using WIN32_LEAN_AND_MEAN macro
## Enabling stringification
By default, Catch does not stringify some types from the standard library. This is done to avoid dragging in various standard library headers by default. However, Catch does contain these and can be configured to provide them, using these macros:
CATCH_CONFIG_ENABLE_PAIR_STRINGMAKER // Provide StringMaker specialization for std::pair
CATCH_CONFIG_ENABLE_TUPLE_STRINGMAKER // Provide StringMaker specialization for std::tuple
CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER // Provide StringMaker specialization for std::variant, std::monostate (on C++17)
CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER // Provide StringMaker specialization for std::optional (on C++17)
CATCH_CONFIG_ENABLE_ALL_STRINGMAKERS // Defines all of the above
> `CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1380) in Catch 2.4.1.
> `CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1510) in Catch 2.6.0.
## Disabling exceptions
> Introduced in Catch 2.4.0.
By default, Catch2 uses exceptions to signal errors and to abort tests
when an assertion from the `REQUIRE` family of assertions fails. We also
provide an experimental support for disabling exceptions. Catch2 should
automatically detect when it is compiled with exceptions disabled, but
it can be forced to compile without exceptions by defining
CATCH_CONFIG_DISABLE_EXCEPTIONS
Note that when using Catch2 without exceptions, there are 2 major
limitations:
1) If there is an error that would normally be signalled by an exception,
the exception's message will instead be written to `Catch::cerr` and
`std::terminate` will be called.
2) If an assertion from the `REQUIRE` family of macros fails,
`std::terminate` will be called after the active reporter returns.
There is also a customization point for the exact behaviour of what
happens instead of exception being thrown. To use it, define
CATCH_CONFIG_DISABLE_EXCEPTIONS_CUSTOM_HANDLER
and provide a definition for this function:
```cpp
namespace Catch {
[[noreturn]]
void throw_exception(std::exception const&);
}
```
## Overriding Catch's debug break (`-b`)
> [Introduced](https://github.com/catchorg/Catch2/pull/1846) in Catch 2.11.2.
You can override Catch2's break-into-debugger code by defining the
`CATCH_BREAK_INTO_DEBUGGER()` macro. This can be used if e.g. Catch2 does
not know your platform, or your platform is misdetected.
The macro will be used as is, that is, `CATCH_BREAK_INTO_DEBUGGER();`
mustcompileandmustbreakintodebugger.
All C++11 support can be disabled with `CATCH_CONFIG_NO_CPP11`
So you want to contribute something to Catch? That's great! Whether it's a bug fix, a new feature, support for additional compilers - or just a fix to the documentation - all contributions are very welcome and very much appreciated. Of course so are bug reports and other comments and questions.
**Contents**<br>
[Branches](#branches)<br>
[Directory structure](#directory-structure)<br>
[Testing your changes](#testing-your-changes)<br>
[Documenting your code](#documenting-your-code)<br>
[Code constructs to watch out for](#code-constructs-to-watch-out-for)<br>
If you are contributing to the code base there are a few simple guidelines to keep in mind. This also includes notes to help you find your way around. As this is liable to drift out of date please raise an issue or, better still, a pull request for this file, if you notice that.
So you want to contribute something to Catch? That's great! Whether it's a bug fix, a new feature, support for
additional compilers - or just a fix to the documentation - all contributions are very welcome and very much appreciated.
Of course so are bug reports and other comments and questions.
If you are contributing to the code base there are a few simple guidelines to keep in mind. This also includes notes to
help you find your way around. As this is liable to drift out of date please raise an issue or, better still, a pull
request for this file, if you notice that.
## Branches
Ongoing development is on the "develop" branch (if there is one, currently), or on feature branches that are branched off of develop. Please target any pull requests at develop, or, for larger chunks of work, a branch off of develop.
Ongoing development is currently on _master_. At some point an integration branch will be set-up and PRs should target
that - but for now it's all against master. You may see feature branches come and go from time to time, too.
## Directory structure
Users of Catch primarily use the single header version. Maintainers should work with the full source (which is still, primarily, in headers). This can be found in the ```include``` folder, but you may prefer to use one of the IDE project files (for MSVC or XCode). These can be found under ```projects/```*IDE Name*```/```*project name*. A number of contributors have proposed make files, and submitted their own versions. At some point these should be made available too.
_Users_ of Catch primarily use the single header version. _Maintainers_ should work with the full source (which is still,
primarily, in headers). This can be found in the `include` folder. There are a set of test files, currently under
`projects/SelfTest`. The test app can be built via CMake from the `CMakeLists.txt` file in the root, or you can generate
project files for Visual Studio, XCode, and others (instructions in the `projects` folder). If you have access to CLion,
it can work with the CMake file directly.
In addition to the include files and IDE projects there are a number of tests in cpp files. These can all be found in ```projects/SelfTest```. You'll also see a ```SurrogateCpps``` directory in there. This contains a set of cpp files that each ```#include``` a single header. While these files are not essential to compilation they help to keep the implementation headers self-contained. At time of writing this set is not complete but has reasonable coverage. If you add additional headers please try to remember to add a surrogate cpp for it.
As well as the runtime test files you'll also see a `SurrogateCpps` directory under `projects/SelfTest`.
This contains a set of .cpp files that each `#include` a single header.
While these files are not essential to compilation they help to keep the implementation headers self-contained.
At time of writing this set is not complete but has reasonable coverage.
If you add additional headers please try to remember to add a surrogate cpp for it.
The other directories are ```scripts``` which contains a set of python scripts to help in testing Catch as well as generating the single include, and docs, which contains the documentation as a set of markdown files.
The other directories are `tools/scripts` which contains a set of python scripts to help in testing Catch as well as
generating the single include, and `docs`, which contains the documentation as a set of markdown files.
__When submitting a pull request please do not include changes to the single include, or to the version number file
as these are managed by the scripts!__
## Testing your changes
Obviously all changes to Catch's code should be tested. If you added new
functionality, you should add tests covering and showcasing it. Even if you have
only made changes to Catch internals (i.e. you implemented some performance
improvements), you should still test your changes.
Over time, some limitations of Catch2 emerged. Some of these are due
to implementation details that cannot be easily changed, some of these
are due to lack of development resources on our part, and some of these
are due to plain old 3rd party bugs.
## Implementation limits
### Sections nested in loops
If you are using `SECTION`s inside loops, you have to create them with
different name per loop's iteration. The recommended way to do so is to
incorporate the loop's counter into section's name, like so:
```cpp
TEST_CASE("Looped section"){
for(chari='0';i<'5';++i){
SECTION(std::string("Looped section ")+i){
SUCCEED("Everything is OK");
}
}
}
```
or with a `DYNAMIC_SECTION` macro (that was made for exactly this purpose):
```cpp
TEST_CASE("Looped section"){
for(chari='0';i<'5';++i){
DYNAMIC_SECTION("Looped section "<<i){
SUCCEED("Everything is OK");
}
}
}
```
### Tests might be run again if last section fails
If the last section in a test fails, it might be run again. This is because
Catch2 discovers `SECTION`s dynamically, as they are about to run, and
if the last section in test case is aborted during execution (e.g. via
the `REQUIRE` family of macros), Catch2 does not know that there are no
more sections in that test case and must run the test case again.
### MinGW/CygWin compilation (linking) is extremely slow
Compiling Catch2 with MinGW can be exceedingly slow, especially during
the linking step. As far as we can tell, this is caused by deficiencies
in its default linker. If you can tell MinGW to instead use lld, via
`-fuse-ld=lld`, the link time should drop down to reasonable length
again.
## Features
This section outlines some missing features, what is their status and their possible workarounds.
### Thread safe assertions
Catch2's assertion macros are not thread safe. This does not mean that
you cannot use threads inside Catch's test, but that only single thread
can interact with Catch's assertions and other macros.
This means that this is ok
```cpp
std::vector<std::thread>threads;
std::atomic<int>cnt{0};
for(inti=0;i<4;++i){
threads.emplace_back([&](){
++cnt;++cnt;++cnt;++cnt;
});
}
for(auto&t:threads){t.join();}
REQUIRE(cnt==16);
```
because only one thread passes the `REQUIRE` macro and this is not
```cpp
std::vector<std::thread>threads;
std::atomic<int>cnt{0};
for(inti=0;i<4;++i){
threads.emplace_back([&](){
++cnt;++cnt;++cnt;++cnt;
CHECK(cnt==16);
});
}
for(auto&t:threads){t.join();}
REQUIRE(cnt==16);
```
Because C++11 provides the necessary tools to do this, we are planning
to remove this limitation in the future.
### Process isolation in a test
Catch does not support running tests in isolated (forked) processes. While this might in the future, the fact that Windows does not support forking and only allows full-on process creation and the desire to keep code as similar as possible across platforms, mean that this is likely to take significant development time, that is not currently available.
### Running multiple tests in parallel
Catch's test execution is strictly serial. If you find yourself with a test suite that takes too long to run and you want to make it parallel, there are 2 feasible solutions
* You can split your tests into multiple binaries and then run these binaries in parallel.
* You can have Catch list contained test cases and then run the same test binary multiple times in parallel, passing each instance list of test cases it should run.
Both of these solutions have their problems, but should let you wring parallelism out of your test suite.
## 3rd party bugs
This section outlines known bugs in 3rd party components (this means compilers, standard libraries, standard runtimes).
### Visual Studio 2017 -- raw string literal in assert fails to compile
There is a known bug in Visual Studio 2017 (VC 15), that causes compilation error when preprocessor attempts to stringize a raw string literal (`#` preprocessor is applied to it). This snippet is sufficient to trigger the compilation error:
```cpp
#define CATCH_CONFIG_MAIN
#include"catch.hpp"
TEST_CASE("test"){
CHECK(std::string(R"("\)") == "\"\\");
}
```
Catch provides a workaround, it is possible to disable stringification of original expressions by defining `CATCH_CONFIG_DISABLE_STRINGIFICATION`:
```cpp
#define CATCH_CONFIG_FAST_COMPILE
#define CATCH_CONFIG_DISABLE_STRINGIFICATION
#include"catch.hpp"
TEST_CASE("test"){
CHECK(std::string(R"("\)") == "\"\\");
}
```
_Do note that this changes the output somewhat_
```
catchwork\test1.cpp(6):
PASSED:
CHECK( Disabled by CATCH_CONFIG_DISABLE_STRINGIFICATION )
with expansion:
""\" == ""\"
```
### Visual Studio 2015 -- Alignment compilation error (C2718)
VS 2015 has a known bug, where `declval<T>` can cause compilation error
if `T` has alignment requirements that it cannot meet.
A workaround is to explicitly specialize `Catch::is_range` for given
type (this avoids code path that uses `declval<T>` in a SFINAE context).
### Visual Studio 2015 -- Wrong line number reported in debug mode
VS 2015 has a known bug where `__LINE__` macro can be improperly expanded under certain circumstances, while compiling multi-file project in Debug mode.
A workaround is to compile the binary in Release mode.
### Clang/G++ -- skipping leaf sections after an exception
Some versions of `libc++` and `libstdc++` (or their runtimes) have a bug with `std::uncaught_exception()` getting stuck returning `true` after rethrow, even if there are no active exceptions. One such case is this snippet, which skipped the sections "a" and "b", when compiled against `libcxxrt` from master
```cpp
#define CATCH_CONFIG_MAIN
#include<catch.hpp>
TEST_CASE("a"){
CHECK_THROWS(throw3);
}
TEST_CASE("b"){
inti=0;
SECTION("a"){i=1;}
SECTION("b"){i=2;}
CHECK(i>0);
}
```
If you are seeing a problem like this, i.e. a weird test paths that trigger only under Clang with `libc++`, or only under very specific version of `libstdc++`, it is very likely you are seeing this. The only known workaround is to use a fixed version of your standard library.
This is a bug in `libstdc++-4.8`, where all matching methods from `<regex>` return false. Since `Matches` uses `<regex>` internally, if the underlying implementation does not work, it doesn't work either.
Workaround: Use newer version of `libstdc++`.
### libstdc++, `_GLIBCXX_DEBUG` macro and random ordering of tests
Running a Catch2 binary compiled against libstdc++ with `_GLIBCXX_DEBUG`
macro defined with `--order rand` will cause a debug check to trigger and
abort the run due to self-assignment.
[This is a known bug inside libstdc++](https://stackoverflow.com/questions/22915325/avoiding-self-assignment-in-stdshuffle/23691322)
Workaround: Don't use `--order rand` when compiling against debug-enabled
Additional messages can be logged during a test case.
Additional messages can be logged during a test case. Note that the messages logged with `INFO` are scoped and thus will not be reported if failure occurs in scope preceding the message declaration. An example:
```cpp
TEST_CASE("Foo"){
INFO("Test case start");
for(inti=0;i<2;++i){
INFO("The number is "<<i);
CHECK(i==0);
}
}
TEST_CASE("Bar"){
INFO("Test case start");
for(inti=0;i<2;++i){
INFO("The number is "<<i);
CHECK(i==i);
}
CHECK(false);
}
```
When the `CHECK` fails in the "Foo" test case, then two messages will be printed.
```
Test case start
The number is 1
```
When the last `CHECK` fails in the "Bar" test case, then only one message will be printed: `Test case start`.
## Logging without local scope
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch 2.7.0.
`UNSCOPED_INFO` is similar to `INFO` with two key differences:
- Lifetime of an unscoped message is not tied to its own scope.
- An unscoped message can be reported by the first following assertion only, regardless of the result of that assertion.
In other words, lifetime of `UNSCOPED_INFO` is limited by the following assertion (or by the end of test case/section, whichever comes first) whereas lifetime of `INFO` is limited by its own scope.
These differences make this macro useful for reporting information from helper functions or inner scopes. An example:
```cpp
voidprint_some_info(){
UNSCOPED_INFO("Info from helper");
}
TEST_CASE("Baz"){
print_some_info();
for(inti=0;i<2;++i){
UNSCOPED_INFO("The number is "<<i);
}
CHECK(false);
}
TEST_CASE("Qux"){
INFO("First info");
UNSCOPED_INFO("First unscoped info");
CHECK(false);
INFO("Second info");
UNSCOPED_INFO("Second unscoped info");
CHECK(false);
}
```
"Baz" test case prints:
```
Info from helper
The number is 0
The number is 1
```
With "Qux" test case, two messages will be printed when the first `CHECK` fails:
```
First info
First unscoped info
```
"First unscoped info" message will be cleared after the first `CHECK`, while "First info" message will persist until the end of the test case. Therefore, when the second `CHECK` fails, three messages will be printed:
```
First info
Second info
Second unscoped info
```
## Streaming macros
All these macros allow heterogenous sequences of values to be streaming using the insertion operator (```<<```) in the same way that std::ostream, std::cout, etc support it.
All these macros allow heterogeneous sequences of values to be streaming using the insertion operator (```<<```) in the same way that std::ostream, std::cout, etc support it.
E.g.:
```c++
@@ -16,7 +99,16 @@ These macros come in three forms:
**INFO(** _message expression_ **)**
The message is logged to a buffer, but only reported with the next assertion that is logged. This allows you to log contextual information in case of failures which is not shown during a successful test run (for the console reporter, without -s). Messages are removed from the buffer at the end of their scope, so may be used, for example, in loops.
The message is logged to a buffer, but only reported with next assertions that are logged. This allows you to log contextual information in case of failures which is not shown during a successful test run (for the console reporter, without -s). Messages are removed from the buffer at the end of their scope, so may be used, for example, in loops.
_Note that in Catch2 2.x.x `INFO` can be used without a trailing semicolon as there is a trailing semicolon inside macro.
This semicolon will be removed with next major version. It is highly advised to use a trailing semicolon after `INFO` macro._
**UNSCOPED_INFO(** _message expression_ **)**
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch 2.7.0.
Similar to `INFO`, but messages are not limited to their own scope: They are removed from the buffer after each assertion, section or test case, whichever comes first.
**WARN(** _message expression_ **)**
@@ -26,27 +118,42 @@ The message is always reported but does not fail the test.
The message is reported and the test case fails.
## Quickly capture a variable value
**FAIL_CHECK(** _message expression_ **)**
**CAPTURE(** _expression_ **)**
AS `FAIL`, but does not abort the test
Sometimes you just want to log the name and value of a variable. While you can easily do this with the INFO macro, above, as a convenience the CAPTURE macro handles the stringising of the variable name for you (actually it works with any expression, not just variables).
## Quickly capture value of variables or expressions
Catch is great for open source. With its [liberal license](../LICENSE.txt) and single-header, dependency-free, distribution
it's easy to just drop the header into your project and start writing tests - what's not to like?
As a result Catch is now being used in many Open Source projects, including some quite well known ones.
This page is an attempt to track those projects. Obviously it can never be complete.
This effort largely relies on the maintainers of the projects themselves updating this page and submitting a PR
(or, if you prefer contact one of the maintainers of Catch directly, use the
[forums](https://groups.google.com/forum/?fromgroups#!forum/catch-forum)), or raise an [issue](https://github.com/philsquared/Catch/issues) to let us know).
Of course users of those projects might want to update this page too. That's fine - as long you're confident the project maintainers won't mind.
If you're an Open Source project maintainer and see your project listed here but would rather it wasn't -
just let us know via any of the previously mentioned means - although I'm sure there won't be many who feel that way.
Listing a project here does not imply endorsement and the plan is to keep these ordered alphabetically to avoid an implication of relative importance.
A, header-only, embedded scripting language designed from the ground up to directly target C++ and take advantage of modern C++ development techniques.
The next-generation core storage and query engine for Couchbase Lite.
### [cppcodec](https://github.com/tplgy/cppcodec)
Header-only C++11 library to encode/decode base64, base64url, base32, base32hex and hex (a.k.a. base16) as specified in RFC 4648, plus Crockford's base32.
A single-header-only library written in C++14 to glue distributed software components (UDP, TCP, shared memory) supporting natively Protobuf, LCM/ZCM, MsgPack, and JSON for dynamic message transformations in-between.
[Let Catch take full control of args and config](#let-catch-take-full-control-of-args-and-config)<br>
[Amending the config](#amending-the-config)<br>
[Adding your own command line options](#adding-your-own-command-line-options)<br>
[Version detection](#version-detection)<br>
The easiest way to use Catch is to let it supply ```main()``` for you and handle configuring itself from the command line.
This is achieved by writing ```#define CATCH_CONFIG_MAIN``` before the ```#include "catch.hpp"``` in *exactly one* source file.
@@ -16,8 +23,7 @@ If you just need to have code that executes before and/ or after Catch this is t
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
int main( int argc, char* const argv[] )
{
int main( int argc, char* argv[] ) {
// global setup...
int result = Catch::Session().run( argc, argv );
@@ -30,28 +36,33 @@ int main( int argc, char* const argv[] )
## Amending the config
If you still want Catch to process the command line, but you want to programatically tweak the config, you can do so in one of two ways:
If you still want Catch to process the command line, but you want to programmatically tweak the config, you can do so in one of two ways:
```c++
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
int main( int argc, char* const argv[] )
int main( int argc, char* argv[] )
{
Catch::Session session; // There must be exactly once instance
Catch::Session session; // There must be exactly one instance
// writing to session.configData() here sets defaults
// this is the preferred way to set them
int returnCode = session.applyCommandLine( argc, argv );
if( returnCode != 0 ) // Indicates a command line error
return returnCode;
return returnCode;
// writing to session.configData() or session.Config() here
// overrides command line args
// only do this if you know you need to
return session.run();
int numFailed = session.run();
// numFailed is clamped to 255 as some unices only use the lower 8 bits.
// This clamping has already been applied, so just return it here
// You can also do any post run clean-up here
return numFailed;
}
```
@@ -61,8 +72,60 @@ To take full control of the config simply omit the call to ```applyCommandLine()
## Adding your own command line options
Catch embeds a powerful command line parser which you can also use to parse your own options out. This capability is still in active development but will be documented here when it is ready.
Catch embeds a powerful command line parser called [Clara](https://github.com/philsquared/Clara).
As of Catch2 (and Clara 1.0) Clara allows you to write _composable_ option and argument parsers,
so extending Catch's own command line options is now easy.
```c++
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
int main( int argc, char* argv[] )
{
Catch::Session session; // There must be exactly one instance
int height = 0; // Some user variable you want to be able to set
// Build a new parser on top of Catch's
using namespace Catch::clara;
auto cli
= session.cli() // Get Catch's composite command line parser
| Opt( height, "height" ) // bind variable to a new option, with a hint string
["-g"]["--height"] // the option names it will respond to
("how high?"); // description string for the help output
// Now pass the new composite back to Catch so it uses that
session.cli( cli );
// Let Catch (using Clara) parse the command line
int returnCode = session.applyCommandLine( argc, argv );
if( returnCode != 0 ) // Indicates a command line error
return returnCode;
// if set on the command line then 'height' is now set at this point
if( height > 0 )
std::cout << "height: " << height << std::endl;
return session.run();
}
```
See the [Clara documentation](https://github.com/philsquared/Clara/blob/master/README.md) for more details.
## Version detection
Catch provides a triplet of macros providing the header's version,
* `CATCH_VERSION_MAJOR`
* `CATCH_VERSION_MINOR`
* `CATCH_VERSION_PATCH`
these macros expand into a single number, that corresponds to the appropriate
part of the version. As an example, given single header version v2.3.4,
the macros would expand into `2`, `3`, and `4` respectively.
When enough changes have accumulated, it is time to release new version of Catch. This document describes the process in doing so, that no steps are forgotten. Note that all referenced scripts can be found in the `tools/scripts/` directory.
## Necessary steps
These steps are necessary and have to be performed before each new release. They serve to make sure that the new release is correct and linked-to from the standard places.
### Testing
All of the tests are currently run in our CI setup based on TravisCI and
AppVeyor. As long as the last commit tested green, the release can
proceed.
### Incrementing version number
Catch uses a variant of [semantic versioning](http://semver.org/), with breaking API changes (and thus major version increments) being very rare. Thus, the release will usually increment the patch version, when it only contains couple of bugfixes, or minor version, when it contains new functionality, or larger changes in implementation of current functionality.
After deciding which part of version number should be incremented, you can use one of the `*Release.py` scripts to perform the required changes to Catch.
This will take care of generating the single include header, updating
version numbers everywhere and pushing the new version to Wandbox.
### Release notes
Once a release is ready, release notes need to be written. They should summarize changes done since last release. For rough idea of expected notes see previous releases. Once written, release notes should be added to `docs/release-notes.md`.
### Commit and push update to GitHub
After version number is incremented, single-include header is regenerated and release notes are updated, changes should be committed and pushed to GitHub.
### Release on GitHub
After pushing changes to GitHub, GitHub release *needs* to be created.
Tag version and release title should be same as the new version,
description should contain the release notes for the current release.
Single header version of `catch.hpp`*needs* to be attached as a binary,
as that is where the official download link links to. Preferably
it should use linux line endings. All non-bundled reporters (Automake, TAP,
TeamCity, SonarQube) should also be attached as binaries, as they might be
dependent on a specific version of the single-include header.
Since 2.5.0, the release tag and the "binaries" (headers) should be PGP
signed.
#### Signing a tag
To create a signed tag, use `git tag -s <VERSION>`, where `<VERSION>`
is the version being released, e.g. `git tag -s v2.6.0`.
Use the version name as the short message and the release notes as
the body (long) message.
#### Signing the headers
This will create ASCII-armored signatures for the headers that are
Catch has a modular reporting system and comes bundled with a handful of useful reporters built in.
You can also write your own reporters.
## Using different reporters
The reporter to use can easily be controlled from the command line.
To specify a reporter use [`-r` or `--reporter`](command-line.md#choosing-a-reporter-to-use), followed by the name of the reporter, e.g.:
```
-r xml
```
If you don't specify a reporter then the console reporter is used by default.
There are four reporters built in to the single include:
*`console` writes as lines of text, formatted to a typical terminal width, with colours if a capable terminal is detected.
*`compact` similar to `console` but optimised for minimal output - each entry on one line
*`junit` writes xml that corresponds to Ant's [junitreport](http://help.catchsoftware.com/display/ET/JUnit+Format) target. Useful for build systems that understand Junit.
Because of the way the junit format is structured the run must complete before anything is written.
*`xml` writes an xml format tailored to Catch. Unlike `junit` this is a streaming format so results are delivered progressively.
There are a few additional reporters, for specific build systems, in the Catch repository (in `include\reporters`) which you can `#include` in your project if you would like to make use of them.
Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`.
*`teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands.
Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)).
*`tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format.
*`automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files
*`sonarqube` writes the [SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format.
You see what reporters are available from the command line by running with `--list-reporters`.
By default all these reports are written to stdout, but can be redirected to a file with [`-o` or `--out`](command-line.md#sending-output-to-a-file)
## Writing your own reporter
You can write your own custom reporter and register it with Catch.
At time of writing the interface is subject to some changes so is not, yet, documented here.
If you are determined you shouldn't have too much trouble working it out from the existing implementations -
but do keep in mind upcoming changes (these will be minor, simplifying, changes such as not needing to forward calls to the base class).
[Other possible solutions](#other-possible-solutions)<br>
Several people have reported that test code written with Catch takes much longer to compile than they would expect. Why is that?
Catch is implemented entirely in headers. There is a little overhead due to this - but not as much as you might think - and you can minimise it simply by organising your test code as follows:
@@ -15,8 +22,51 @@ But functions and methods can also be written inline in header files. The downsi
Because Catch is implemented *entirely* in headers you might think that the whole of Catch must be compiled into every translation unit that uses it! Actually it's not quite as bad as that. Catch mitigates this situation by effectively maintaining the traditional separation between the implementation code and declarations. Internally the implementation code is protected by ```#ifdef```s and is conditionally compiled into only one translation unit. This translation unit is that one that ```#define```s ```CATCH_CONFIG_MAIN``` or ```CATCH_CONFIG_RUNNER```. Let's call this the main source file.
As a result the main source file *does* compile the whole of Catch every time! So it makes sense to dedicate this file to *only* ```#define```-ing the identifier and ```#include```-ing Catch (and implementing the runner code, if you're doing that). Keep all your test cases in other files. This way you won't pay the recompilation cost for the whole of Catch
As a result the main source file *does* compile the whole of Catch every time! So it makes sense to dedicate this file to *only* ```#define```-ing the identifier and ```#include```-ing Catch (and implementing the runner code, if you're doing that). Keep all your test cases in other files. This way you won't pay the recompilation cost for the whole of Catch.
## Practical example
Assume you have the `Factorial` function from the [tutorial](tutorial.md#top) in `factorial.cpp` (with forward declaration in `factorial.h`) and want to test it and keep the compile times down when adding new tests. Then you should have 2 files, `tests-main.cpp` and `tests-factorial.cpp`:
```cpp
// tests-main.cpp
#define CATCH_CONFIG_MAIN
#include "catch.hpp"
```
```cpp
// tests-factorial.cpp
#include "catch.hpp"
#include "factorial.h"
TEST_CASE( "Factorials are computed", "[factorial]" ) {
REQUIRE( Factorial(1) == 1 );
REQUIRE( Factorial(2) == 2 );
REQUIRE( Factorial(3) == 6 );
REQUIRE( Factorial(10) == 3628800 );
}
```
After compiling `tests-main.cpp` once, it is enough to link it with separately compiled `tests-factorial.cpp`. This means that adding more tests to `tests-factorial.cpp`, will not result in recompiling Catch's main and the resulting compilation times will decrease substantially.
Now, the next time we change the file `tests-factorial.cpp` (say we add `REQUIRE( Factorial(0) == 1)`), it is enough to recompile the tests instead of recompiling main as well:
You can also opt to sacrifice some features in order to speed-up Catch's compilation times. For details see the [documentation on Catch's compile-time configuration](configuration.md#other-toggles).
[Type parametrised test cases](#type-parametrised-test-cases)<br>
[Signature based parametrised test cases](#signature-based-parametrised-test-cases)<br>
While Catch fully supports the traditional, xUnit, style of class-based fixtures containing test case methods this is not the preferred style.
Instead Catch provides a powerful mechanism for nesting test case sections within a test case. For a more detailed discussion see the [tutorial](tutorial.md#testCasesAndSections).
Instead Catch provides a powerful mechanism for nesting test case sections within a test case. For a more detailed discussion see the [tutorial](tutorial.md#test-cases-and-sections).
Test cases and sections are very easy to use in practice:
_test name_ and _section name_ are free form, quoted, strings. The optional _tags_ argument is a quoted string containing one or more tags enclosed in square brackets. Tags are discussed below. Test names must be unique within the Catch executable.
For examples see the [Tutorial](tutorial.md)
_test name_ and _section name_ are free form, quoted, strings.
The optional _tags_ argument is a quoted string containing one or more
tags enclosed in square brackets, and are discussed below.
_section description_ can be used to provide long form description
of a section while keeping the _section name_ short for use with the
[`-c` command line parameter](command-line.md#specify-the-section-to-run).
**Test names must be unique within the Catch executable.**
For examples see the [Tutorial](tutorial.md#top)
## Tags
@@ -19,42 +35,46 @@ Tags allow an arbitrary number of additional strings to be associated with a tes
The tag expression, ```"[widget]"``` selects A, B & D. ```"[gadget]"``` selects C & D. ```"[widget][gadget]"``` selects just D and ```"[widget],[gadget]"``` selects all four test cases.
For more detail on command line selection see [the command line docs](command-line.md#specifying-which-tests-to-run)
Tag names are not case sensitive.
Tag names are not case sensitive and can contain any ASCII characters. This means that tags `[tag with spaces]` and `[I said "good day"]` are both allowed tags and can be filtered on. Escapes are not supported however and `[\]]` is not a valid tag.
### Special Tags
All tag names beginning with non-alphanumeric characters are reserved by Catch. Catch defines a number of "special" tags, which have meaning to the test runner itself. These special tags all begin with a symbol character. Following is a list of currently defined special tags and their meanings.
* `[!hide]` or `[.]` (or, for legacy reasons, `[hide]`) - causes test cases to be skipped from the default list (ie when no test cases have been explicitly selected through tag expressions or name wildcards). The hide tag is often combined with another, user, tag (for example `[.][integration]` - so all integration tests are excluded from the default run but can be run by passing `[integration]` on the command line). As a short-cut you can combine these by simply prefixing your user tag with a `.` - e.g. `[.integration]`. Because the hide tag has evolved to have several forms, all forms are added as tags if you use one of them.
* `[.]` - causes test cases to be skipped from the default list (i.e. when no test cases have been explicitly selected through tag expressions or name wildcards). The hide tag is often combined with another, user, tag (for example `[.][integration]` - so all integration tests are excluded from the default run but can be run by passing `[integration]` on the command line). As a short-cut you can combine these by simply prefixing your user tag with a `.` - e.g. `[.integration]`.
* `[!throws]`- lets Catch know that this test is likely to throw an exception even if successful. This causes the test to be exluded when running with `-e` or `--nothrow`.
* `[!throws]`- lets Catch know that this test is likely to throw an exception even if successful. This causes the test to be excluded when running with `-e` or `--nothrow`.
* `[!shouldfail]` - reverse the failing logic of the test: if the test is successful if it fails, and vice-versa.
* `[!mayfail]` - doesn't fail the test if any given assertion fails (but still reports it). This can be useful to flag a work-in-progress, or a known issue that you don't want to immediately fix but still want to track in your tests.
* `[!mayfail]` - doesn't fail the test if any given assertion fails (but still reports it). This can be useful to flag a work-in-progress, or a known issue that you don't want to immediately fix but still want to track in the your tests.
* `[!shouldfail]` - like `[!mayfail]` but *fails* the test if it *passes*. This can be useful if you want to be notified of accidental, or third-party, fixes.
* `[#<filename>]` - running with `-#` or `--filenames-as-tags` causes Catch to add the filename, prefixed with `#` (and with any extension stripped) as a tag. e.g. tests in testfile.cpp would all be tagged `[#testfile]`.
* `[!nonportable]` - Indicates that behaviour may vary between platforms or compilers.
* `[#<filename>]` - running with `-#` or `--filenames-as-tags` causes Catch to add the filename, prefixed with `#` (and with any extension stripped), as a tag to all contained tests, e.g. tests in testfile.cpp would all be tagged `[#testfile]`.
* `[@<alias>]` - tag aliases all begin with `@` (see below).
* `[!benchmark]` - this test case is actually a benchmark. This is an experimental feature, and currently has no documentation. If you want to try it out, look at `projects/SelfTest/Benchmark.tests.cpp` for details.
## Tag aliases
Between tag expressions and wildcarded test names (as well as combinations of the two) quite complex patterns can be constructed to direct which test cases are run. If a complex pattern is used often it is convenient to be able to create an alias for the expression. this can be done, in code, using the following form:
Between tag expressions and wildcarded test names (as well as combinations of the two) quite complex patterns can be constructed to direct which test cases are run. If a complex pattern is used often it is convenient to be able to create an alias for the expression. This can be done, in code, using the following form:
Now when `[@nhf]` is used on the command line this matches all tests that are tagged `[failing]`, but which are not also hidden.
@@ -72,15 +92,192 @@ This macro maps onto ```TEST_CASE``` and works in the same way, except that the
These macros map onto ```SECTION```s except that the section names are the _something_s prefixed by "given: ", "when: " or "then: " respectively.
* **AND_GIVEN(** _something_ **)**
* **AND_WHEN(** _something_ **)**
* **AND_THEN(** _something_ **)**
Similar to ```WHEN``` and ```THEN``` except that the prefixes start with "and ". These are used to chain ```WHEN```s and ```THEN```s together.
Similar to ```GIVEN```, ```WHEN``` and ```THEN``` except that the prefixes start with "and ". These are used to chain ```GIVEN```s, ```WHEN```s and ```THEN```s together.
> `AND_GIVEN` was [introduced](https://github.com/catchorg/Catch2/issues/1360) in Catch 2.4.0.
When any of these macros are used the console reporter recognises them and formats the test case header such that the Givens, Whens and Thens are aligned to aid readability.
Other than the additional prefixes and the formatting in the console reporter these macros behave exactly as ```TEST_CASE```s and ```SECTION```s. As such there is nothing enforcing the correct sequencing of these macros - that's up to the programmer!
## Type parametrised test cases
In addition to `TEST_CASE`s, Catch2 also supports test cases parametrised
by types, in the form of `TEMPLATE_TEST_CASE`,
`TEMPLATE_PRODUCT_TEST_CASE` and `TEMPLATE_LIST_TEST_CASE`.
Although Catch allows you to group tests together as sections within a test case, it can still convenient, sometimes, to group them using a more traditional test fixture. Catch fully supports this too. You define the test fixture as a simple structure:
<aid="top"></a>
# Test fixtures
## Defining test fixtures
Although Catch allows you to group tests together as sections within a test case, it can still be convenient, sometimes, to group them using a more traditional test fixture. Catch fully supports this too. You define the test fixture as a simple structure:
```c++
class UniqueTestsFixture {
@@ -27,6 +32,112 @@ class UniqueTestsFixture {
The two test cases here will create uniquely-named derived classes of UniqueTestsFixture and thus can access the `getID()` protected method and `conn` member variables. This ensures that both the test cases are able to create a DBConnection using the same method (DRY principle) and that any ID's created are unique such that the order that tests are executed does not matter.
Catch2 also provides `TEMPLATE_TEST_CASE_METHOD` and
`TEMPLATE_PRODUCT_TEST_CASE_METHOD` that can be used together
with templated fixtures and templated template fixtures to perform
tests for multiple different types. Unlike `TEST_CASE_METHOD`,
`TEMPLATE_TEST_CASE_METHOD` and `TEMPLATE_PRODUCT_TEST_CASE_METHOD` do
require the tag specification to be non-empty, as it is followed by
further macro arguments.
Also note that, because of limitations of the C++ preprocessor, if you
want to specify a type with multiple template parameters, you need to
enclose it in parentheses, e.g. `std::map<int,std::string>` needs to be
passed as `(std::map<int,std::string>)`.
In the case of `TEMPLATE_PRODUCT_TEST_CASE_METHOD`, if a member of the
type list should consist of more than single type, it needs to be enclosed
in another pair of parentheses, e.g. `(std::map, std::pair)` and
`((int, float), (char, double))`.
Example:
```cpp
template< typename T >
struct Template_Fixture {
Template_Fixture(): m_a(1) {}
T m_a;
};
TEMPLATE_TEST_CASE_METHOD(Template_Fixture,"A TEMPLATE_TEST_CASE_METHOD based test run that succeeds", "[class][template]", int, float, double) {
REQUIRE( Template_Fixture<TestType>::m_a == 1 );
}
template<typename T>
struct Template_Template_Fixture {
Template_Template_Fixture() {}
T m_a;
};
template<typename T>
struct Foo_class {
size_t size() {
return 0;
}
};
TEMPLATE_PRODUCT_TEST_CASE_METHOD(Template_Template_Fixture, "A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test succeeds", "[class][template]", (Foo_class, std::vector), int) {
_While there is an upper limit on the number of types you can specify
in single `TEMPLATE_TEST_CASE_METHOD` or `TEMPLATE_PRODUCT_TEST_CASE_METHOD`,
the limit is very high and should not be encountered in practice._
## Signature-based parametrised test fixtures
> [Introduced](https://github.com/catchorg/Catch2/issues/1609) in Catch 2.8.0.
Catch2 also provides `TEMPLATE_TEST_CASE_METHOD_SIG` and `TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG` to support
fixtures using non-type template parameters. These test cases work similar to `TEMPLATE_TEST_CASE_METHOD` and `TEMPLATE_PRODUCT_TEST_CASE_METHOD`,
with additional positional argument for [signature](test-cases-and-sections.md#signature-based-parametrised-test-cases).
Example:
```cpp
template <int V>
struct Nttp_Fixture{
int value = V;
};
TEMPLATE_TEST_CASE_METHOD_SIG(Nttp_Fixture, "A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds", "[class][template][nttp]",((int V), V), 1, 3, 6) {
REQUIRE(Nttp_Fixture<V>::value > 0);
}
template<typename T>
struct Template_Fixture_2 {
Template_Fixture_2() {}
T m_a;
};
template< typename T, size_t V>
struct Template_Foo_2 {
size_t size() { return V; }
};
TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG(Template_Fixture_2, "A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds", "[class][template][product][nttp]", ((typename T, size_t S), T, S),(std::array, Template_Foo_2), ((int,2), (float,6)))
## Template fixtures with types specified in template type lists
Catch2 also provides `TEMPLATE_LIST_TEST_CASE_METHOD` to support template fixtures with types specified in
template type lists like `std::tuple`, `boost::mpl::list` or `boost::mp11::mp_list`. This test case works the same as `TEMPLATE_TEST_CASE_METHOD`,
only difference is the source of types. This allows you to reuse the template type list in multiple test cases.
Example:
```cpp
using MyTypes = std::tuple<int, char, double>;
TEMPLATE_LIST_TEST_CASE_METHOD(Template_Fixture, "Template test case method with test types specified inside std::tuple", "[class][template][list]", MyTypes)
[Floating point precision](#floating-point-precision)<br>
Catch needs to be able to convert types you use in assertions and logging expressions into strings (for logging and reporting purposes).
Most built-in or std types are supported out of the box but there are three ways that you can tell Catch how to convert your own types (or other, third-party types) into strings.
Most built-in or std types are supported out of the box but there are two ways that you can tell Catch how to convert your own types (or other, third-party types) into strings.
## operator << overload for std::ostream
This is the standard way of providing string conversions in C++ - and the chances are you may already provide this for your own purposes. If you're not familiar with this idiom it involves writing a free function of the form:
(where ```T``` is your type and ```convertMyTypeToString``` is where you'll write whatever code is necessary to make your type printable - it doesn't have to be in another function).
You should put this function in the same namespace as your type.
You should put this function in the same namespace as your type, or the global namespace, and have it declared before including Catch's header.
Alternatively you may prefer to write it as a member function:
## Catch::StringMaker specialisation
If you don't want to provide an ```operator <<``` overload, or you want to convert your type differently for testing purposes, you can provide a specialization for `Catch::StringMaker<T>`:
```
std::ostream& T::operator << ( std::ostream& os ) const {
os << convertMyTypeToString( *this );
return os;
}
```
## Catch::toString overload
If you don't want to provide an ```operator <<``` overload, or you want to convert your type differently for testing purposes, you can provide an overload for ```Catch::toString()``` for your type.
```
```cpp
namespace Catch {
std::string toString( T const& value ) {
return convertMyTypeToString( value );
}
template<>
struct StringMaker<T> {
static std::string convert( T const& value ) {
return convertMyTypeToString( value );
}
};
}
```
Again ```T``` is your type and ```convertMyTypeToString``` is where you'll write whatever code is necessary to make your type printable. Note that the function must be in the Catch namespace, which itself must be in the global namespace.
## Catch::is_range specialisation
As a fallback, Catch attempts to detect if the type can be iterated
(`begin(T)` and `end(T)` are valid) and if it can be, it is stringified
as a range. For certain types this can lead to infinite recursion, so
it can be disabled by specializing `Catch::is_range` like so:
## Catch::StringMaker<T> specialisation
There are some cases where overloading toString does not work as expected. Specialising StringMaker<T> gives you more precise, and reliable, control - but at the cost of slightly more code and complexity:
```
```cpp
namespace Catch {
template<> struct StringMaker<T> {
static std::string convert( T const& value ) {
return convertMyTypeToString( value );
}
};
template<>
struct is_range<T> {
static const bool value = false;
};
}
```
## Exceptions
By default all exceptions deriving from `std::exception` will be translated to strings by calling the `what()` method. For exception types that do not derive from `std::exception` - or if `what()` does not return a suitable string - use `CATCH_TRANSLATE_EXCEPTION`. This defines a function that takes your exception type, by reference, and returns a string. It can appear anywhere in the code - it doesn't have to be in the same translation unit. For example:
```
```cpp
CATCH_TRANSLATE_EXCEPTION( MyType& ex ) {
return ex.message();
return ex.message();
}
```
## Enums
> Introduced in Catch 2.8.0.
Enums that already have a `<<` overload for `std::ostream` will convert to strings as expected.
If you only need to convert enums to strings for test reporting purposes you can provide a `StringMaker` specialisations as any other type.
However, as a convenience, Catch provides the `REGISTER_ENUM` helper macro that will generate the `StringMaker` specialiation for you with minimal code.
Simply provide it the (qualified) enum name, followed by all the enum values, and you're done!
The simplest way to get Catch is to download the single header version from [http://builds.catch-lib.net](http://builds.catch-lib.net). Don't be put off by the word "builds" there. The single header is generated by merging a set of individual headers but it is still just normal source code in a header file.
**Contents**<br>
[Getting Catch2](#getting-catch2)<br>
[Where to put it?](#where-to-put-it)<br>
[Writing tests](#writing-tests)<br>
[Test cases and sections](#test-cases-and-sections)<br>
[BDD-Style](#bdd-style)<br>
[Scaling up](#scaling-up)<br>
[Type parametrised test cases](#type-parametrised-test-cases)<br>
[Next steps](#next-steps)<br>
The full source for Catch, including test projects, documentation, and other things, is hosted on GitHub. [http://catch-lib.net](http://catch-lib.net) will redirect you there.
## Getting Catch2
The simplest way to get Catch2 is to download the latest [single header version](https://raw.githubusercontent.com/catchorg/Catch2/master/single_include/catch2/catch.hpp). The single header is generated by merging a set of individual headers but it is still just normal source code in a header file.
Alternative ways of getting Catch2 include using your system package
manager, or installing it using [its CMake package](cmake-integration.md#installing-catch2-from-git-repository).
The full source for Catch2, including test projects, documentation, and other things, is hosted on GitHub. [http://catch-lib.net](http://catch-lib.net) will redirect you there.
## Where to put it?
Catch is header only. All you need to do is drop the file(s) somewhere reachable from your project - either in some central location you can set your header search path to find, or directly into your project tree itself! This is a particularly good option for other Open-Source projects that want to use Catch for their test suite. See [this blog entry for more on that](http://www.levelofindirection.com/journal/2011/5/27/unit-testing-in-c-and-objective-c-just-got-ridiculously-easi.html).
Catch2 is header only. All you need to do is drop the file somewhere reachable from your project - either in some central location you can set your header search path to find, or directly into your project tree itself! This is a particularly good option for other Open-Source projects that want to use Catch for their test suite. See [this blog entry for more on that](https://levelofindirection.com/blog/unit-testing-in-cpp-and-objective-c-just-got-ridiculously-easier-still.html).
The rest of this tutorial will assume that the Catch single-include header (or the include folder) is available unqualified - but you may need to prefix it with a folder name if necessary.
The rest of this tutorial will assume that the Catch2 single-include header (or the include folder) is available unqualified - but you may need to prefix it with a folder name if necessary.
# Writing tests
_If you have installed Catch2 from system package manager, or CMake
package, you need to include the header as `#include <catch2/catch.hpp>`_
Let's start with a really simple example. Say you have written a function to calculate factorials and now you want to test it (let's leave aside TDD for now).
## Writing tests
Let's start with a really simple example ([code](../examples/010-TestCase.cpp)). Say you have written a function to calculate factorials and now you want to test it (let's leave aside TDD for now).
```c++
unsigned int Factorial( unsigned int number ) {
@@ -21,7 +40,7 @@ unsigned int Factorial( unsigned int number ) {
}
```
To keep things simple we'll put everything in a single file (<a href="#scaling-up">see later for more on how to structure your test files</a>)
To keep things simple we'll put everything in a single file (<a href="#scaling-up">see later for more on how to structure your test files</a>).
```c++
#define CATCH_CONFIG_MAIN // This tells Catch to provide a main() - only do this in one cpp file
This will compile to a complete executable which responds to [command line arguments](command-line.md). If you just run it with no arguments it will execute all test cases (in this case there is just one), report any failures, report a summary of how many tests passed and failed and return the number of failed tests (useful for if you just want a yes/ no answer to: "did it work").
This will compile to a complete executable which responds to [command line arguments](command-line.md#top). If you just run it with no arguments it will execute all test cases (in this case there is just one), report any failures, report a summary of how many tests passed and failed and return the number of failed tests (useful for if you just want a yes/ no answer to: "did it work").
If you run this as written it will pass. Everything is good. Right?
Well, there is still a bug here. In fact the first version of this tutorial I posted here genuinely had the bug in! So it's not completely contrived (thanks to Daryle Walker (```@CTMacUser```) for pointing this out).
@@ -68,7 +87,7 @@ with expansion:
0 == 1
```
Note that we get the actual return value of Factorial(0) printed for us (0) - even though we used a natural expression with the == operator. That let's us immediately see what the problem is.
Note that we get the actual return value of Factorial(0) printed for us (0) - even though we used a natural expression with the == operator. That lets us immediately see what the problem is.
Let's change the factorial function to:
@@ -80,14 +99,14 @@ unsigned int Factorial( unsigned int number ) {
Now all the tests pass.
Of course there are still more issues to do deal with. For example we'll hit problems when the return value starts to exceed the range of an unsigned int. With factorials that can happen quite quickly. You might want to add tests for such cases and decide how to handle them. We'll stop short of doing that here.
Of course there are still more issues to deal with. For example we'll hit problems when the return value starts to exceed the range of an unsigned int. With factorials that can happen quite quickly. You might want to add tests for such cases and decide how to handle them. We'll stop short of doing that here.
## What did we do here?
### What did we do here?
Although this was a simple test it's been enough to demonstrate a few things about how Catch is used. Let's take moment to consider those before we move on.
Although this was a simple test it's been enough to demonstrate a few things about how Catch is used. Let's take a moment to consider those before we move on.
1. All we did was ```#define``` one identifier and ```#include``` one header and we got everything - even an implementation of ```main()``` that will [respond to command line arguments](command-line.md). You can only use that ```#define``` in one implementation file, for (hopefully) obvious reasons. Once you have more than one file with unit tests in you'll just ```#include "catch.hpp"``` and go. Usually it's a good idea to have a dedicated implementation file that just has ```#define CATCH_CONFIG_MAIN``` and ```#include "catch.hpp"```. You can also provide your own implementation of main and drive Catch yourself (see [Supplying-your-own-main()](own-main.md)).
2. We introduce test cases with the ```TEST_CASE``` macro. This macro takes one or two arguments - a free form test name and, optionally, one or more tags (for more see <a href="#test-cases-and-sections">Test cases and Sections</a>, ). The test name must be unique. You can run sets of tests by specifying a wildcarded test name or a tag expression. See the [command line docs](command-line.md) for more information on running tests.
1. All we did was ```#define``` one identifier and ```#include``` one header and we got everything - even an implementation of ```main()``` that will [respond to command line arguments](command-line.md#top). You can only use that ```#define``` in one implementation file, for (hopefully) obvious reasons. Once you have more than one file with unit tests in you'll just ```#include "catch.hpp"``` and go. Usually it's a good idea to have a dedicated implementation file that just has ```#define CATCH_CONFIG_MAIN``` and ```#include "catch.hpp"```. You can also provide your own implementation of main and drive Catch yourself (see [Supplying-your-own-main()](own-main.md#top)).
2. We introduce test cases with the ```TEST_CASE``` macro. This macro takes one or two arguments - a free form test name and, optionally, one or more tags (for more see <a href="#test-cases-and-sections">Test cases and Sections</a>). The test name must be unique. You can run sets of tests by specifying a wildcarded test name or a tag expression. See the [command line docs](command-line.md#top) for more information on running tests.
3. The name and tags arguments are just strings. We haven't had to declare a function or method - or explicitly register the test case anywhere. Behind the scenes a function with a generated name is defined for you, and automatically registered using static registry classes. By abstracting the function name away we can name our tests without the constraints of identifier names.
4. We write our individual test assertions using the ```REQUIRE``` macro. Rather than a separate macro for each type of condition we express the condition naturally using C/C++ syntax. Behind the scenes a simple set of expression templates captures the left-hand-side and right-hand-side of the expression so we can display the values in our test report. As we'll see later there _are_ other assertion macros - but because of this technique the number of them is drastically reduced.
@@ -98,37 +117,37 @@ Most test frameworks have a class-based fixture mechanism. That is, test cases m
While Catch fully supports this way of working there are a few problems with the approach. In particular the way your code must be split up, and the blunt granularity of it, may cause problems. You can only have one setup/ teardown pair across a set of methods, but sometimes you want slightly different setup in each method, or you may even want several levels of setup (a concept which we will clarify later on in this tutorial). It was <a href="http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html">problems like these</a> that led James Newkirk, who led the team that built NUnit, to start again from scratch and <a href="http://jamesnewkirk.typepad.com/posts/2007/09/announcing-xuni.html">build xUnit</a>).
Catch takes a different approach (to both NUnit and xUnit) that is a more natural fit for C++ and the C family of languages. This is best explained through an example:
Catch takes a different approach (to both NUnit and xUnit) that is a more natural fit for C++ and the C family of languages. This is best explained through an example ([code](../examples/100-Fix-Section.cpp)):
```c++
TEST_CASE( "vectors can be sized and resized", "[vector]" ) {
std::vector<int> v( 5 );
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 5 );
SECTION( "resizing bigger changes size and capacity" ) {
v.resize( 10 );
REQUIRE( v.size() == 10 );
REQUIRE( v.capacity() >= 10 );
}
SECTION( "resizing smaller changes size but not capacity" ) {
v.resize( 0 );
REQUIRE( v.size() == 0 );
REQUIRE( v.capacity() >= 5 );
}
SECTION( "reserving bigger changes capacity but not size" ) {
v.reserve( 10 );
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 10 );
}
SECTION( "reserving smaller does not change size or capacity" ) {
v.reserve( 0 );
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 5 );
}
@@ -140,18 +159,18 @@ This works because the ```SECTION``` macro contains an if statement that calls b
So far so good - this is already an improvement on the setup/teardown approach because now we see our setup code inline and use the stack.
The power of sections really shows, however, when we need to execute a sequence of, checked, operations. Continuing the vector example, we might want to verify that attempting to reserve a capacity smaller than the current capacity of the vector changes nothing. We can do that, naturally, like so:
The power of sections really shows, however, when we need to execute a sequence of checked operations. Continuing the vector example, we might want to verify that attempting to reserve a capacity smaller than the current capacity of the vector changes nothing. We can do that, naturally, like so:
```c++
SECTION( "reserving bigger changes capacity but not size" ) {
v.reserve( 10 );
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 10 );
SECTION( "reserving smaller again does not change capacity" ) {
v.reserve( 7 );
REQUIRE( v.capacity() >= 10 );
}
}
@@ -161,22 +180,22 @@ Sections can be nested to an arbitrary depth (limited only by your stack size).
## BDD-Style
If you name your test cases and sections appropriately you can achieve a BDD-style specification structure. This became such a useful way of working that first class support has been added to Catch. Scenarios can be specified using ```SCENARIO```, ```GIVEN```, ```WHEN``` and ```THEN``` macros, which map on to ```TEST_CASE```s and ```SECTION```s, respectively. For more details see [Test cases and sections](test-cases-and-sections.md).
If you name your test cases and sections appropriately you can achieve a BDD-style specification structure. This became such a useful way of working that first class support has been added to Catch. Scenarios can be specified using ```SCENARIO```, ```GIVEN```, ```WHEN``` and ```THEN``` macros, which map on to ```TEST_CASE```s and ```SECTION```s, respectively. For more details see [Test cases and sections](test-cases-and-sections.md#top).
The vector example can be adjusted to use these macros like so:
The vector example can be adjusted to use these macros like so ([example code](../examples/120-Bdd-ScenarioGivenWhenThen.cpp)):
```c++
SCENARIO( "vectors can be sized and resized", "[vector]" ) {
GIVEN( "A vector with some items" ) {
std::vector<int> v( 5 );
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 5 );
WHEN( "the size is increased" ) {
v.resize( 10 );
THEN( "the size and capacity change" ) {
REQUIRE( v.size() == 10 );
REQUIRE( v.capacity() >= 10 );
@@ -184,7 +203,7 @@ SCENARIO( "vectors can be sized and resized", "[vector]" ) {
}
WHEN( "the size is reduced" ) {
v.resize( 0 );
THEN( "the size changes but not capacity" ) {
REQUIRE( v.size() == 0 );
REQUIRE( v.capacity() >= 5 );
@@ -192,7 +211,7 @@ SCENARIO( "vectors can be sized and resized", "[vector]" ) {
}
WHEN( "more capacity is reserved" ) {
v.reserve( 10 );
THEN( "the capacity changes but not the size" ) {
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 10 );
@@ -200,7 +219,7 @@ SCENARIO( "vectors can be sized and resized", "[vector]" ) {
}
WHEN( "less capacity is reserved" ) {
v.reserve( 0 );
THEN( "neither size nor capacity are changed" ) {
REQUIRE( v.size() == 5 );
REQUIRE( v.capacity() >= 5 );
@@ -224,7 +243,7 @@ Scenario: vectors can be sized and resized
To keep the tutorial simple we put all our code in a single file. This is fine to get started - and makes jumping into Catch even quicker and easier. As you write more real-world tests, though, this is not really the best approach.
The requirement is that the following block of code ([or equivalent](own-main.md)):
The requirement is that the following block of code ([or equivalent](own-main.md#top)):
```c++
#define CATCH_CONFIG_MAIN
@@ -233,17 +252,28 @@ The requirement is that the following block of code ([or equivalent](own-main.md
appears in _exactly one_ source file. Use as many additional cpp files (or whatever you call your implementation files) as you need for your tests, partitioned however makes most sense for your way of working. Each additional file need only ```#include "catch.hpp"``` - do not repeat the ```#define```!
In fact it is usually a good idea to put the block with the ```#define``` [in it's own source file](slow-compiles.md).
In fact it is usually a good idea to put the block with the ```#define``` [in its own source file](slow-compiles.md#top) (code example [main](../examples/020-TestCase-1.cpp), [tests](../examples/020-TestCase-2.cpp)).
Do not write your tests in header files!
## Type parametrised test cases
Test cases in Catch2 can be also parametrised by type, via the
`TEMPLATE_TEST_CASE` and `TEMPLATE_PRODUCT_TEST_CASE` macros,
which behave in the same way the `TEST_CASE` macro, but are run for
every type or type combination.
For more details, see our documentation on [test cases and
This has been a brief introduction to get you up and running with Catch, and to point out some of the key differences between Catch and other frameworks you may already be familiar with. This will get you going quite far already and you are now in a position to dive in and write some tests.
Of course there is more to learn - most of which you should be able to page-fault in as you go. Please see the ever-growing [Reference section](Readme.md) for what's available.
Of course there is more to learn - most of which you should be able to page-fault in as you go. Please see the ever-growing [Reference section](Readme.md#top) for what's available.
Good question. For C++ there are quite a number of established frameworks, including (but not limited to), [CppUnit](http://sourceforge.net/apps/mediawiki/cppunit/index.php?title=Main_Page), [Google Test](http://code.google.com/p/googletest/), [Boost.Test](http://www.boost.org/doc/libs/1_49_0/libs/test/doc/html/index.html), [Aeryn](https://launchpad.net/aeryn), [Cute](http://r2.ifs.hsr.ch/cute), [Fructose](http://fructose.sourceforge.net/) and [many, many more](http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B). Even for Objective-C there are a few, including OCUnit - which now comes bundled with XCode.
Good question. For C++ there are quite a number of established frameworks,
[many, many more](http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B).
So what does Catch bring to the party that differentiates it from these? Apart from a Catchy name, of course.
## Key Features
* Really easy to get started. Just download catch.hpp, #include it and you're away.
* No external dependencies. As long as you can compile C++98 and have a C++ standard library available.
* Write test cases as, self-registering, functions or methods.
* Divide test cases into sections, each of which is run in isolation (eliminates the need for fixtures!)
* Quick and Really easy to get started. Just download catch.hpp, `#include` it and you're away.
* No external dependencies. As long as you can compile C++11 and have a C++ standard library available.
* Write test cases as, self-registering, functions (or methods, if you prefer).
* Divide test cases into sections, each of which is run in isolation (eliminates the need for fixtures).
* Use BDD-style Given-When-Then sections as well as traditional unit test cases.
* Only one core assertion macro for comparisons. Standard C/C++ operators are used for the comparison - yet the full expression is decomposed and lhs and rhs values are logged.
* Tests are named using free-form strings - no more couching names in legal identifiers.
## Other core features
* Tests are named using free-form strings - no more couching names in legal identifiers.
* Tests can be tagged for easily running ad-hoc groups of tests.
* Failures can (optionally) break into the debugger on Windows and Mac.
* Output is through modular reporter objects. Basic textual and XML reporters are included. Custom reporters can easily be added.
* JUnit xml output is supported for integration with third-party tools, such as CI servers.
* A default main() function is provided (in a header), but you can supply your own for complete control (e.g. integration into your own test runner GUI).
* A default main() function is provided, but you can supply your own for complete control (e.g. integration into your own test runner GUI).
* A command line parser is provided and can still be used if you choose to provided your own main() function.
* Catch can test itself.
* Alternative assertion macro(s) report failures but don't abort the test case
* Floating point tolerance comparisons are built in using an expressive Approx() syntax.
* Internal and friendly macros are isolated so name clashes can be managed
*Support for Matchers (early stages)
*Matchers
## Objective-C-specific features
## Who else is using Catch?
* Automatically detects if you are using it from an Objective-C project
* Works with and without ARC with no additional configuration
* Implement test fixtures using Obj-C classes too (like OCUnit)
* Additional built in matchers that work with Obj-C types (e.g. string matchers)
See the list of [open source projects using Catch](opensource-users.md#top).
See the [tutorial](tutorial.md) to get more of a taste of using CATCH in practice
See the [tutorial](tutorial.md#top) to get more of a taste of using Catch in practice
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.