mirror of
https://github.com/catchorg/Catch2.git
synced 2024-11-22 05:16:10 +01:00
parent
6d803cba5d
commit
037ddbc75c
@ -1,7 +1,7 @@
|
||||
<a id="top"></a>
|
||||
# Authoring benchmarks
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
|
||||
|
||||
Writing benchmarks is not easy. Catch simplifies certain aspects but you'll
|
||||
always need to take care about various aspects. Understanding a few things about
|
||||
|
@ -226,7 +226,7 @@ When set to ```yes``` Catch will report the duration of each test case, in milli
|
||||
|
||||
<pre>-D, --min-duration <value></pre>
|
||||
|
||||
> `--min-duration` was [introduced](https://github.com/catchorg/Catch2/pull/1910) in Catch 2.13.0
|
||||
> `--min-duration` was [introduced](https://github.com/catchorg/Catch2/pull/1910) in Catch2 2.13.0
|
||||
|
||||
When set, Catch will report the duration of each test case that took more
|
||||
than <value> seconds, in milliseconds. This option is overriden by both
|
||||
@ -302,7 +302,7 @@ either before running any tests, after running all tests - or both, depending on
|
||||
## Specify the number of benchmark samples to collect
|
||||
<pre>--benchmark-samples <# of samples></pre>
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
|
||||
|
||||
When running benchmarks a number of "samples" is collected. This is the base data for later statistical analysis.
|
||||
Per sample a clock resolution dependent number of iterations of the user code is run, which is independent of the number of samples. Defaults to 100.
|
||||
@ -311,7 +311,7 @@ Per sample a clock resolution dependent number of iterations of the user code is
|
||||
## Specify the number of resamples for bootstrapping
|
||||
<pre>--benchmark-resamples <# of resamples></pre>
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
|
||||
|
||||
After the measurements are performed, statistical [bootstrapping] is performed
|
||||
on the samples. The number of resamples for that bootstrapping is configurable
|
||||
@ -326,7 +326,7 @@ defaults to 95%).
|
||||
## Specify the confidence-interval for bootstrapping
|
||||
<pre>--benchmark-confidence-interval <confidence-interval></pre>
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
|
||||
|
||||
The confidence-interval is used for statistical bootstrapping on the samples to
|
||||
calculate the upper and lower bounds of mean and standard deviation.
|
||||
@ -336,7 +336,7 @@ Must be between 0 and 1 and defaults to 0.95.
|
||||
## Disable statistical analysis of collected benchmark samples
|
||||
<pre>--benchmark-no-analysis</pre>
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1616) in Catch2 2.9.0.
|
||||
|
||||
When this flag is specified no bootstrapping or any other statistical analysis is performed.
|
||||
Instead the user code is only measured and the plain mean from the samples is reported.
|
||||
@ -345,7 +345,7 @@ Instead the user code is only measured and the plain mean from the samples is re
|
||||
## Specify the amount of time in milliseconds spent on warming up each test
|
||||
<pre>--benchmark-warmup-time</pre>
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/pull/1844) in Catch 2.11.2.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/pull/1844) in Catch2 2.11.2.
|
||||
|
||||
Configure the amount of time spent warming up each test.
|
||||
|
||||
|
@ -117,7 +117,7 @@ Catch's selection, by defining either `CATCH_CONFIG_CPP11_TO_STRING` or
|
||||
CATCH_CONFIG_CPP17_OPTIONAL // Override std::optional support detection (checked by CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER)
|
||||
CATCH_CONFIG_CPP17_BYTE // Override std::byte support detection (Catch provides a StringMaker specialization by default)
|
||||
|
||||
> `CATCH_CONFIG_CPP17_STRING_VIEW` was [introduced](https://github.com/catchorg/Catch2/issues/1376) in Catch 2.4.1.
|
||||
> `CATCH_CONFIG_CPP17_STRING_VIEW` was [introduced](https://github.com/catchorg/Catch2/issues/1376) in Catch2 2.4.1.
|
||||
|
||||
Catch contains basic compiler/standard detection and attempts to use
|
||||
some C++17 features whenever appropriate. This automatic detection
|
||||
@ -141,7 +141,7 @@ by using `_NO_` in the macro, e.g. `CATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS`.
|
||||
CATCH_CONFIG_ANDROID_LOGWRITE // Use android's logging system for debug output
|
||||
CATCH_CONFIG_GLOBAL_NEXTAFTER // Use nextafter{,f,l} instead of std::nextafter
|
||||
|
||||
> [`CATCH_CONFIG_ANDROID_LOGWRITE`](https://github.com/catchorg/Catch2/issues/1743) and [`CATCH_CONFIG_GLOBAL_NEXTAFTER`](https://github.com/catchorg/Catch2/pull/1739) were introduced in Catch 2.10.0
|
||||
> [`CATCH_CONFIG_ANDROID_LOGWRITE`](https://github.com/catchorg/Catch2/issues/1743) and [`CATCH_CONFIG_GLOBAL_NEXTAFTER`](https://github.com/catchorg/Catch2/pull/1739) were introduced in Catch2 2.10.0
|
||||
|
||||
Currently Catch enables `CATCH_CONFIG_WINDOWS_SEH` only when compiled with MSVC, because some versions of MinGW do not have the necessary Win32 API support.
|
||||
|
||||
@ -197,13 +197,13 @@ By default, Catch does not stringify some types from the standard library. This
|
||||
CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER // Provide StringMaker specialization for std::optional (on C++17)
|
||||
CATCH_CONFIG_ENABLE_ALL_STRINGMAKERS // Defines all of the above
|
||||
|
||||
> `CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1380) in Catch 2.4.1.
|
||||
> `CATCH_CONFIG_ENABLE_VARIANT_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1380) in Catch2 2.4.1.
|
||||
|
||||
> `CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1510) in Catch 2.6.0.
|
||||
> `CATCH_CONFIG_ENABLE_OPTIONAL_STRINGMAKER` was [introduced](https://github.com/catchorg/Catch2/issues/1510) in Catch2 2.6.0.
|
||||
|
||||
## Disabling exceptions
|
||||
|
||||
> Introduced in Catch 2.4.0.
|
||||
> Introduced in Catch2 2.4.0.
|
||||
|
||||
By default, Catch2 uses exceptions to signal errors and to abort tests
|
||||
when an assertion from the `REQUIRE` family of assertions fails. We also
|
||||
@ -239,7 +239,7 @@ namespace Catch {
|
||||
|
||||
## Overriding Catch's debug break (`-b`)
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/pull/1846) in Catch 2.11.2.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/pull/1846) in Catch2 2.11.2.
|
||||
|
||||
You can override Catch2's break-into-debugger code by defining the
|
||||
`CATCH_BREAK_INTO_DEBUGGER()` macro. This can be used if e.g. Catch2 does
|
||||
|
@ -147,9 +147,9 @@ be replaced with the actual version upon release. There are 2 styles
|
||||
of placeholders used through the documentation, you should pick one that
|
||||
fits your text better (if in doubt, take a look at the existing version
|
||||
tags for other features).
|
||||
* `> [Introduced](link-to-issue-or-PR) in Catch X.Y.Z` - this
|
||||
* `> [Introduced](link-to-issue-or-PR) in Catch2 X.Y.Z` - this
|
||||
placeholder is usually used after a section heading
|
||||
* `> X (Y and Z) was [introduced](link-to-issue-or-PR) in Catch X.Y.Z`
|
||||
* `> X (Y and Z) was [introduced](link-to-issue-or-PR) in Catch2 X.Y.Z`
|
||||
- this placeholder is used when you need to tag a subpart of something,
|
||||
e.g. a list
|
||||
|
||||
|
@ -1,7 +1,7 @@
|
||||
<a id="top"></a>
|
||||
# Data Generators
|
||||
|
||||
> Introduced in Catch 2.6.0.
|
||||
> Introduced in Catch2 2.6.0.
|
||||
|
||||
Data generators (also known as _data driven/parametrized test cases_)
|
||||
let you reuse the same set of assertions across different input values.
|
||||
@ -92,7 +92,7 @@ TEST_CASE("Complex mix of sections and generates") {
|
||||
}
|
||||
```
|
||||
|
||||
> The ability to place `GENERATE` between two `SECTION`s was [introduced](https://github.com/catchorg/Catch2/issues/1938) in Catch 2.13.0.
|
||||
> The ability to place `GENERATE` between two `SECTION`s was [introduced](https://github.com/catchorg/Catch2/issues/1938) in Catch2 2.13.0.
|
||||
|
||||
## Provided generators
|
||||
|
||||
@ -117,9 +117,9 @@ a test case,
|
||||
* `RangeGenerator<T>` -- generates all values inside an arithmetic range
|
||||
* `IteratorGenerator<T>` -- copies and returns values from an iterator range
|
||||
|
||||
> `ChunkGenerator<T>`, `RandomIntegerGenerator<Integral>`, `RandomFloatGenerator<Float>` and `RangeGenerator<T>` were introduced in Catch 2.7.0.
|
||||
> `ChunkGenerator<T>`, `RandomIntegerGenerator<Integral>`, `RandomFloatGenerator<Float>` and `RangeGenerator<T>` were introduced in Catch2 2.7.0.
|
||||
|
||||
> `IteratorGenerator<T>` was introduced in Catch 2.10.0.
|
||||
> `IteratorGenerator<T>` was introduced in Catch2 2.10.0.
|
||||
|
||||
The generators also have associated helper functions that infer their
|
||||
type, making their usage much nicer. These are
|
||||
@ -139,11 +139,11 @@ type, making their usage much nicer. These are
|
||||
* `from_range(InputIterator from, InputIterator to)` for `IteratorGenerator<T>`
|
||||
* `from_range(Container const&)` for `IteratorGenerator<T>`
|
||||
|
||||
> `chunk()`, `random()` and both `range()` functions were introduced in Catch 2.7.0.
|
||||
> `chunk()`, `random()` and both `range()` functions were introduced in Catch2 2.7.0.
|
||||
|
||||
> `from_range` has been introduced in Catch 2.10.0
|
||||
> `from_range` has been introduced in Catch2 2.10.0
|
||||
|
||||
> `range()` for floating point numbers has been introduced in Catch 2.11.0
|
||||
> `range()` for floating point numbers has been introduced in Catch2 2.11.0
|
||||
|
||||
And can be used as shown in the example below to create a generator
|
||||
that returns 100 odd random number:
|
||||
@ -176,7 +176,7 @@ scope and thus capturing references is dangerous. If you need to use
|
||||
variables inside the generator expression, make sure you thought through
|
||||
the lifetime implications and use `GENERATE_COPY` or `GENERATE_REF`.**
|
||||
|
||||
> `GENERATE_COPY` and `GENERATE_REF` were introduced in Catch 2.7.1.
|
||||
> `GENERATE_COPY` and `GENERATE_REF` were introduced in Catch2 2.7.1.
|
||||
|
||||
You can also override the inferred type by using `as<type>` as the first
|
||||
argument to the macro. This can be useful when dealing with string literals,
|
||||
|
@ -30,7 +30,7 @@ When the last `CHECK` fails in the "Bar" test case, then only one message will b
|
||||
|
||||
## Logging without local scope
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch 2.7.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch2 2.7.0.
|
||||
|
||||
`UNSCOPED_INFO` is similar to `INFO` with two key differences:
|
||||
|
||||
@ -106,7 +106,7 @@ This semicolon will be removed with next major version. It is highly advised to
|
||||
|
||||
**UNSCOPED_INFO(** _message expression_ **)**
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch 2.7.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1522) in Catch2 2.7.0.
|
||||
|
||||
Similar to `INFO`, but messages are not limited to their own scope: They are removed from the buffer after each assertion, section or test case, whichever comes first.
|
||||
|
||||
|
@ -125,7 +125,7 @@ These are
|
||||
* `Equals` which checks whether the result is exactly equal (order matters) to a specific vector
|
||||
* `UnorderedEquals` which checks whether the result is equal to a specific vector under a permutation
|
||||
* `Approx` which checks whether the result is "approx-equal" (order matters, but comparison is done via `Approx`) to a specific vector
|
||||
> Approx matcher was [introduced](https://github.com/catchorg/Catch2/issues/1499) in Catch 2.7.2.
|
||||
> Approx matcher was [introduced](https://github.com/catchorg/Catch2/issues/1499) in Catch2 2.7.2.
|
||||
|
||||
An example usage:
|
||||
```cpp
|
||||
@ -148,7 +148,7 @@ are:
|
||||
* `WithinUlps(FloatingPoint target, uint64_t maxUlpDiff)`, and
|
||||
* `WithinRel(FloatingPoint target, FloatingPoint eps)`.
|
||||
|
||||
> `WithinRel` matcher was introduced in Catch 2.10.0
|
||||
> `WithinRel` matcher was introduced in Catch2 2.10.0
|
||||
|
||||
|
||||
`WithinAbs` creates a matcher that accepts floating point numbers whose
|
||||
@ -243,7 +243,7 @@ Note that `DerivedException` in the example above has to derive from
|
||||
|
||||
### Generic range Matchers
|
||||
|
||||
> Generic range matchers were introduced in Catch X.Y.Z
|
||||
> Generic range matchers were introduced in Catch2 X.Y.Z
|
||||
|
||||
Catch2 also provides some matchers that use the new style matchers
|
||||
definitions to handle generic range-like types. These are:
|
||||
@ -345,7 +345,7 @@ style matchers arbitrarily.
|
||||
|
||||
## Writing custom matchers (new style)
|
||||
|
||||
> New style matchers were introduced in Catch X.Y.Z
|
||||
> New style matchers were introduced in Catch2 X.Y.Z
|
||||
|
||||
To create a new-style matcher, you have to create your own type that
|
||||
derives from `Catch::Matchers::MatcherGenericBase`. Your type has to
|
||||
|
@ -59,7 +59,7 @@ TEST_CASE( "SUCCEED showcase" ) {
|
||||
|
||||
* `STATIC_REQUIRE`
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1362) in Catch 2.4.2.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1362) in Catch2 2.4.2.
|
||||
|
||||
`STATIC_REQUIRE( expr )` is a macro that can be used the same way as a
|
||||
`static_assert`, but also registers the success with Catch2, so it is
|
||||
@ -134,7 +134,7 @@ ANON_TEST_CASE() {
|
||||
|
||||
* `DYNAMIC_SECTION`
|
||||
|
||||
> Introduced in Catch 2.3.0.
|
||||
> Introduced in Catch2 2.3.0.
|
||||
|
||||
`DYNAMIC_SECTION` is a `SECTION` where the user can use `operator<<` to
|
||||
create the final name for that section. This can be useful with e.g.
|
||||
|
@ -98,7 +98,7 @@ These macros map onto ```SECTION```s except that the section names are the _some
|
||||
|
||||
Similar to ```GIVEN```, ```WHEN``` and ```THEN``` except that the prefixes start with "and ". These are used to chain ```GIVEN```s, ```WHEN```s and ```THEN```s together.
|
||||
|
||||
> `AND_GIVEN` was [introduced](https://github.com/catchorg/Catch2/issues/1360) in Catch 2.4.0.
|
||||
> `AND_GIVEN` was [introduced](https://github.com/catchorg/Catch2/issues/1360) in Catch2 2.4.0.
|
||||
|
||||
When any of these macros are used the console reporter recognises them and formats the test case header such that the Givens, Whens and Thens are aligned to aid readability.
|
||||
|
||||
@ -112,7 +112,7 @@ by types, in the form of `TEMPLATE_TEST_CASE`,
|
||||
|
||||
* **TEMPLATE_TEST_CASE(** _test name_ , _tags_, _type1_, _type2_, ..., _typen_ **)**
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1437) in Catch 2.5.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1437) in Catch2 2.5.0.
|
||||
|
||||
_test name_ and _tag_ are exactly the same as they are in `TEST_CASE`,
|
||||
with the difference that the tag string must be provided (however, it
|
||||
@ -164,7 +164,7 @@ TEMPLATE_TEST_CASE( "vectors can be sized and resized", "[vector][template]", in
|
||||
|
||||
* **TEMPLATE_PRODUCT_TEST_CASE(** _test name_ , _tags_, (_template-type1_, _template-type2_, ..., _template-typen_), (_template-arg1_, _template-arg2_, ..., _template-argm_) **)**
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1468) in Catch 2.6.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1468) in Catch2 2.6.0.
|
||||
|
||||
_template-type1_ through _template-typen_ is list of template template
|
||||
types which should be combined with each of _template-arg1_ through
|
||||
@ -209,7 +209,7 @@ is very high and should not be encountered in practice._
|
||||
|
||||
* **TEMPLATE_LIST_TEST_CASE(** _test name_, _tags_, _type list_ **)**
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1627) in Catch 2.9.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1627) in Catch2 2.9.0.
|
||||
|
||||
_type list_ is a generic list of types on which test case should be instantiated.
|
||||
List can be `std::tuple`, `boost::mpl::list`, `boost::mp11::mp_list` or anything with
|
||||
@ -229,7 +229,7 @@ TEMPLATE_LIST_TEST_CASE("Template test case with test types specified inside std
|
||||
|
||||
## Signature based parametrised test cases
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1609) in Catch 2.8.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1609) in Catch2 2.8.0.
|
||||
|
||||
In addition to [type parametrised test cases](#type-parametrised-test-cases) Catch2 also supports
|
||||
signature base parametrised test cases, in form of `TEMPLATE_TEST_CASE_SIG` and `TEMPLATE_PRODUCT_TEST_CASE_SIG`.
|
||||
|
@ -88,7 +88,7 @@ the limit is very high and should not be encountered in practice._
|
||||
|
||||
## Signature-based parametrised test fixtures
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1609) in Catch 2.8.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1609) in Catch2 2.8.0.
|
||||
|
||||
Catch2 also provides `TEMPLATE_TEST_CASE_METHOD_SIG` and `TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG` to support
|
||||
fixtures using non-type template parameters. These test cases work similar to `TEMPLATE_TEST_CASE_METHOD` and `TEMPLATE_PRODUCT_TEST_CASE_METHOD`,
|
||||
|
@ -71,7 +71,7 @@ CATCH_TRANSLATE_EXCEPTION( MyType const& ex ) {
|
||||
|
||||
## Enums
|
||||
|
||||
> Introduced in Catch 2.8.0.
|
||||
> Introduced in Catch2 2.8.0.
|
||||
|
||||
Enums that already have a `<<` overload for `std::ostream` will convert to strings as expected.
|
||||
If you only need to convert enums to strings for test reporting purposes you can provide a `StringMaker` specialisations as any other type.
|
||||
@ -110,7 +110,7 @@ TEST_CASE() {
|
||||
|
||||
## Floating point precision
|
||||
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1614) in Catch 2.8.0.
|
||||
> [Introduced](https://github.com/catchorg/Catch2/issues/1614) in Catch2 2.8.0.
|
||||
|
||||
Catch provides a built-in `StringMaker` specialization for both `float`
|
||||
and `double`. By default, it uses what we think is a reasonable precision,
|
||||
|
@ -127,8 +127,8 @@ def updateVersionDefine(version):
|
||||
def updateVersionPlaceholder(filename, version):
|
||||
with open(filename, 'rb') as file:
|
||||
lines = file.readlines()
|
||||
placeholderRegex = re.compile(b' in Catch X.Y.Z')
|
||||
replacement = ' in Catch {}.{}.{}'.format(version.majorVersion, version.minorVersion, version.patchNumber).encode('ascii')
|
||||
placeholderRegex = re.compile(b' in Catch[0-9]? X.Y.Z')
|
||||
replacement = ' in Catch2 {}.{}.{}'.format(version.majorVersion, version.minorVersion, version.patchNumber).encode('ascii')
|
||||
with open(filename, 'wb') as file:
|
||||
for line in lines:
|
||||
file.write(placeholderRegex.sub(replacement, line))
|
||||
|
Loading…
Reference in New Issue
Block a user