From 32e81790d68132d7f1a6afa6f06cf0eddf83c8a0 Mon Sep 17 00:00:00 2001 From: Malcolm Noyes Date: Mon, 18 Nov 2013 22:01:48 +0000 Subject: [PATCH] Added instructions for running tests from the command line --- docs/vs/VS2010commandline.md | 113 +++++++++++ docs/vs/VS2010instructions.md | 4 + docs/vs/VS2012commandline.md | 258 ++++++++++++++++++++++++++ docs/vs/VS2012managed-instructions.md | 4 + docs/vs/VS2012native-instructions.md | 4 + docs/vs/vs-index.md | 6 +- 6 files changed, 388 insertions(+), 1 deletion(-) create mode 100644 docs/vs/VS2010commandline.md create mode 100644 docs/vs/VS2012commandline.md diff --git a/docs/vs/VS2010commandline.md b/docs/vs/VS2010commandline.md new file mode 100644 index 00000000..fed1c5f4 --- /dev/null +++ b/docs/vs/VS2010commandline.md @@ -0,0 +1,113 @@ +VS2010 supports MSTest from the command line: + +``` +C:\Projects\VS2010TestProjectExample>"c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe" /testcontainer:Debug\DefaultTest.dll +Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1 +Copyright (c) Microsoft Corporation. All rights reserved. + +Loading Debug\DefaultTest.dll... +Starting execution... + +Results Top Level Tests +------- --------------- +Failed UnitTest1.?A0x0bae6f70.C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1.C_A_T_C_H___M_E_T_H_O_D___10_1 +Passed UnitTest1.?A0x0bae6f70.C_A_T_C_H____T_E_S_T____C_L_A_S_S___6_0.C_A_T_C_H___M_E_T_H_O_D___6_0 +1/2 test(s) Passed, 1 Failed + +Summary +------- +Test Run Failed. + Failed 1 + Passed 1 + --------- + Total 2 +Results file: C:\Projects\VS2010TestProjectExample\TestResults\user_machine 2013-11-18 21_43_26.trx +Test Settings: Default Test Settings +``` +which gives the following output in the .trx file: + +``` + + + + + + + + + + + + + + + + + My first failing catch test in VS2010 + + + + Description + My first failing catch test in VS2010 + + + + + + My first catch test in VS2010 + + + + Description + My first catch test in VS2010 + + + + + + + + + + + + + + + + + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using Catch v1.0 b13 for a managed MSTest project. +------------------------------------------------------------------------------- +My first failing catch test in VS2010 +------------------------------------------------------------------------------- +UnitTest1.cpp(10) +............................................................................... + +UnitTest1.cpp(12): FAILED: + REQUIRE( false ) + +=============================================================================== +0 test cases - all failed + + Assert.Fail failed. REQUIRE( false ), + at UnitTest1.?A0x0bae6f70.C_A_T_C_H____T_E_S_T____10_1() in c:\projects\vs2010testprojectexample\vs2010testprojectexample\unittest1.cpp:line 12 + at Catch.FreeFunctionTestCase.invoke(FreeFunctionTestCase* ) in c:\projects\phil\catch\include\internal\catch_vs_test_registry.hpp:line 32 + at Catch.TestCase.invoke(TestCase* ) in c:\projects\phil\catch\include\internal\catch_test_case_info.hpp:line 78 + at Catch.RunContext.runCurrentTest(RunContext* , basic_string<char\,std::char_traits<char>\,std::allocator<char> >* redirectedCout, basic_string<char\,std::char_traits<char>\,std::allocator<char> >* redirectedCerr) in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 282 + at Catch.RunContext.runTest(RunContext* , Totals* , TestCase* testCase) in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 122 + at UnitTest1.?A0x0bae6f70.C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1.C_A_T_C_H___M_E_T_H_O_D___10_1() in c:\projects\vs2010testprojectexample\vs2010testprojectexample\unittest1.cpp:line 10 + + + + + + + + +``` + + +--- + +[Home](../../README.md) \ No newline at end of file diff --git a/docs/vs/VS2010instructions.md b/docs/vs/VS2010instructions.md index b4d6cd8e..b41d5151 100644 --- a/docs/vs/VS2010instructions.md +++ b/docs/vs/VS2010instructions.md @@ -45,6 +45,10 @@ Now if you click the link for (in this case) "...unittest1.cpp: line 12" you get And that's it! +## Running tests from the command line + +Tests can also be run from the command line. [For VS2010 see these details](VS2010commandline.md) + ## Creating a Catch console project If you want to run Catch normally from the console, just create a new Windows Console app (you will need to turn off Unicode for this!) and add an additional .cpp file for `'main()'`. I usually add a file called main.cpp, like this: diff --git a/docs/vs/VS2012commandline.md b/docs/vs/VS2012commandline.md new file mode 100644 index 00000000..6298938f --- /dev/null +++ b/docs/vs/VS2012commandline.md @@ -0,0 +1,258 @@ +VS2012 supports MSTest for Managed tests and vstest.console.exe for Native tests. + +## Managed Tests + +To run managed tests from the command line: + +``` +C:\Projects\VS2012ManagedTestExample>"c:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Debug\DefaultTest.dll +Microsoft (R) Test Execution Command Line Tool Version 11.0.50727.1 +Copyright (c) Microsoft Corporation. All rights reserved. + +Loading Debug\DefaultTest.dll... +Starting execution... + +Results Top Level Tests +------- --------------- +Failed UnitTest1.?A0x066a5656.C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1.C_A_T_C_H___M_E_T_H_O_D___10_1 +Passed UnitTest1.?A0x066a5656.C_A_T_C_H____T_E_S_T____C_L_A_S_S___6_0.C_A_T_C_H___M_E_T_H_O_D___6_0 +1/2 test(s) Passed, 1 Failed + +Summary +------- +Test Run Failed. + Failed 1 + Passed 1 + --------- + Total 2 +Results file: C:\Projects\VS2012ManagedTestExample\TestResults\user_machine 2013-11-18 21_50_39.trx +Test Settings: Default Test Settings +``` +which gives the following output in the .trx file: + +``` + + + + + + + + + + + + + + + + + My first catch test in VS2010 + + + + Description + My first catch test in VS2010 + + + + + + My first failing catch test in VS2010 + + + + Description + My first failing catch test in VS2010 + + + + + + + + + + + + + + + + + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using Catch v1.0 b13 for a managed MSTest project. +------------------------------------------------------------------------------- +My first failing catch test in VS2010 +------------------------------------------------------------------------------- +UnitTest.cpp(10) +............................................................................... + +UnitTest.cpp(12): FAILED: + REQUIRE( false ) + +=============================================================================== +0 test cases - all failed + + Assert.Fail failed. REQUIRE( false ), + at UnitTest1.?A0x066a5656.C_A_T_C_H____T_E_S_T____10_1() in c:\projects\vs2012managedtestexample\vs2012managedtestexample\unittest.cpp:line 12 + at Catch.FreeFunctionTestCase.invoke(FreeFunctionTestCase* ) in c:\projects\phil\catch\include\internal\catch_vs_test_registry.hpp:line 32 + at Catch.TestCase.invoke(TestCase* ) in c:\projects\phil\catch\include\internal\catch_test_case_info.hpp:line 78 + at Catch.RunContext.runCurrentTest(RunContext* , basic_string<char\,std::char_traits<char>\,std::allocator<char> >* redirectedCout, basic_string<char\,std::char_traits<char>\,std::allocator<char> >* redirectedCerr) in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 282 + at Catch.RunContext.runTest(RunContext* , Totals* , TestCase* testCase) in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 122 + at UnitTest1.?A0x066a5656.C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1.C_A_T_C_H___M_E_T_H_O_D___10_1() in c:\projects\vs2012managedtestexample\vs2012managedtestexample\unittest.cpp:line 10 + + + + + + + + +``` + +## Native Tests + +Native have to be run using vstest.console.exe: + +``` +C:\Projects\VS2012NativeTestExample>"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Logger:Trx debug\VS2012NativeTestExample.dll +Microsoft (R) Test Execution Command Line Tool Version 11.0.60315.1 +Copyright (c) Microsoft Corporation. All rights reserved. + +Starting test execution, please wait... +Information: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using Catch v1.0 b13 for a native MSTest project. +------------------------------------------------------------------------------- +"My first failing catch test in VS2010" +------------------------------------------------------------------------------- +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(10) +............................................................................... + +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(12): FAILED: + REQUIRE( false ) + +=============================================================================== +0 test cases - all failed + + + +Passed C_A_T_C_H___M_E_T_H_O_D___6_0 +Failed C_A_T_C_H___M_E_T_H_O_D___10_1 +Error Message: + Assert failed. false, . c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp, "My first failing catch test in VS2010" line 12 +Stack Trace: + at UnitTest1::C_A_T_C_H____T_E_S_T____10_1() in c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp:line 12 + at Catch::FreeFunctionTestCase::invoke() in c:\projects\phil\catch\include\internal\catch_vs_test_registry.hpp:line 32 + at Catch::TestCase::invoke() in c:\projects\phil\catch\include\internal\catch_test_case_info.hpp:line 78 + at Catch::RunContext::runCurrentTest() in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 274 + + at Catch::RunContext::runTest() in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 124 + at UnitTest1::C_A_T_C_H___N_S_10_1::C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1::C_A_T_C_H___M_E_T_H_O_D___10_1() in c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp:line 10Standard Output Messages: + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Using Catch v1.0 b13 for a native MSTest project. + ------------------------------------------------------------------------------- + "My first failing catch test in VS2010" + ------------------------------------------------------------------------------- + c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(10) + ............................................................................... + + c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(12): FAILED: + REQUIRE( false ) + + =============================================================================== + 0 test cases - all failed + + +Results File: C:\Projects\VS2012NativeTestExample\TestResults\user_machine 2013-11-18 21_53_40.trx + +Total tests: 2. Passed: 1. Failed: 1. Skipped: 0. +Test Run Failed. +Test execution time: 1.5498 Seconds +``` + +which gives results like this in the .trx file: + +``` + + + + + + + + + + + + + + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using Catch v1.0 b13 for a native MSTest project. +------------------------------------------------------------------------------- +"My first failing catch test in VS2010" +------------------------------------------------------------------------------- +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(10) +............................................................................... + +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(12): FAILED: + REQUIRE( false ) + +=============================================================================== +0 test cases - all failed + + Assert failed. false, . c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp, "My first failing catch test in VS2010" line 12 + at UnitTest1::C_A_T_C_H____T_E_S_T____10_1() in c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp:line 12 + at Catch::FreeFunctionTestCase::invoke() in c:\projects\phil\catch\include\internal\catch_vs_test_registry.hpp:line 32 + at Catch::TestCase::invoke() in c:\projects\phil\catch\include\internal\catch_test_case_info.hpp:line 78 + at Catch::RunContext::runCurrentTest() in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 274 + at Catch::RunContext::runTest() in c:\projects\phil\catch\include\internal\catch_runner_impl.hpp:line 124 + at UnitTest1::C_A_T_C_H___N_S_10_1::C_A_T_C_H____T_E_S_T____C_L_A_S_S___10_1::C_A_T_C_H___M_E_T_H_O_D___10_1() in c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp:line 10 + + + + + + + + + + + + + + + + + + + + + + + + + + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using Catch v1.0 b13 for a native MSTest project. +------------------------------------------------------------------------------- +"My first failing catch test in VS2010" +------------------------------------------------------------------------------- +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(10) +............................................................................... + +c:\projects\vs2012nativetestexample\vs2012nativetestexample\unittest1.cpp(12): FAILED: + REQUIRE( false ) + +=============================================================================== +0 test cases - all failed + + + + + +``` + +--- + +[Home](../../README.md) \ No newline at end of file diff --git a/docs/vs/VS2012managed-instructions.md b/docs/vs/VS2012managed-instructions.md index 0d631c01..bce1fa51 100644 --- a/docs/vs/VS2012managed-instructions.md +++ b/docs/vs/VS2012managed-instructions.md @@ -49,6 +49,10 @@ Now if you click the link at the top of the stack trace "...unittest.cpp: line 1 And that's it! +## Running tests from the command line + +Tests can also be run from the command line. [For VS2012 see these details](VS2012commandline.md) + ## Creating a Catch console project If you want to run Catch normally from the console, just create a new Windows Console app (you will need to turn off Unicode for this!) and add an additional .cpp file for `'main()'`. I usually add a file called main.cpp, like this: diff --git a/docs/vs/VS2012native-instructions.md b/docs/vs/VS2012native-instructions.md index 37e142cb..d7b91f22 100644 --- a/docs/vs/VS2012native-instructions.md +++ b/docs/vs/VS2012native-instructions.md @@ -45,6 +45,10 @@ Now if you click the link at the top of the stack trace "...unittest1.cpp: line And that's it! +## Running tests from the command line + +Tests can also be run from the command line. [For VS2012 see these details](VS2012commandline.md) + ## Creating a Catch console project If you want to run Catch normally from the console, just create a new Windows Console app (you will need to turn off Unicode for this!) and add an additional .cpp file for `'main()'`. I usually add a file called main.cpp, like this: diff --git a/docs/vs/vs-index.md b/docs/vs/vs-index.md index 2e38acac..a117b0d3 100644 --- a/docs/vs/vs-index.md +++ b/docs/vs/vs-index.md @@ -1,4 +1,4 @@ -Catch now supports integration with Visual Studio 2010 and 2012. It does this by redefining the macros to use existing VS features; whilst this isn't perfect, it seems to be a workable solution for those people who want to use the VS IDE, or for those who need to provide integration with CI systems that are able to process the test results generated by using MSBuild/test runner???. +Catch now supports integration with Visual Studio 2010 and 2012. It does this by redefining the macros to use existing VS features; whilst this isn't perfect, it seems to be a workable solution for those people who want to use the VS IDE, or for those who need to provide integration with CI systems that are able to process the test results generated by using MSBuild/vstest.console.exe. What does this mean? It means that (with a little care) you can use the same source code for regular Catch, Managed Catch projects in VS and Native Catch projects in VS2012, and you can run tests, see the results and 'click to go to the error' when a test fails. @@ -18,6 +18,10 @@ We can do the same for VS2012 - [see this page for instructions on how to do thi ...and you'll end up with something like this: ![VS2012 failing test example](VS2012failingtest.png) +# Running tests from the command line + +Tests can also be run from the command line. [For VS2010 see these details](VS2010commandline.md) and [for VS2012 see these](VS2012commandline.md) + # Differences in behaviour There are some minor differences in behaviour between Catch and Visual Studio projects; some are inevitable consequences of the 'run one test at a time' architecture of VS, whilst others are workarounds for 'features' in VS itself: