catchorg
Init / cleanup in tests?
Is there a mechanism to perform initialization before any test cases run, and cleanup after they are done?
How is this being accomplished today?
struct test_init {
test_init() {
// perform your init here
}
} test_init_instance;
That will happen at static initialization time. That might cause issues. It would be better if catch's test execution worked like so:
int main()
{
InitTests();
Run();
CleanupTests();
}
Yes, I know I can implement this myself, but I'm just asking if there is a built-in alternative. I dug through the source code and I didn't see anything but just want to be sure.
Implementing your own main is the documented way. It is how I accomplish the global setup in my test runner.
Sorry I misreaded you (you mentioned "before any test cases") I leave my old answer here:
There's no need for setup teardown:
TEST_CASTE(){
// setup
SECTION(){
}
// TEARDOWN
}
the setup and teardown are implicit in how catch work (wich is what I like most). Of course the same apply to nested cases:
TEST_CASTE(){
// setup A
SECTION(){ // A
// setup B,C
SECTION(){ //B
}
SECTION(){ // C
}
// TEARDOWN B,C
}
// TEARDOWN A
}
Further to @jfemia: https://github.com/philsquared/Catch/blob/develop/docs/own-main.md
A remark about @Darelbi comments. The "teardown" part of a test doesn't always work. On my MS VS2010 machine (catch ver. 1.21.) the following code produces memory leak because "arr" isn't deleted. It leaves the "section A" on REQUIRE exception immediately and doesn't continue to the "delete" part. Oddly though, I see that it enters the "test A" right after that and this time performs new/delete cycle without entering into SECTION.
TEST_CASE("test A")
{
char *arr = new char[8];
SECTION("section A")
{
REQUIRE(1 == 0);
}
delete []arr;
}
So I had to revert to std::unique_ptr
TEST_CASE("test B")
{
std::unique_ptr<char[]> arr(new char[8]);
SECTION("section B")
{
REQUIRE(1 == 0);
}
}
Yes, if you need clean-up within a test case you should use some form of RAII (whether that's with a unique_ptr, scope_guard, or some class of your own design).
@nabijaczleweli I got your irony, but we don't argument about the resource management. The fact is there is no guarantee that the after-section "teardown" part will be always executed.
Honestly I wouldn't attempt to do any kind of cleanup in test case macros. The behavior of how they are executed may change at anytime. The structure of test cases may be misleading.
RAII is the only guaranteed solution.
+1 for "RAII is the only guaranteed solution". The fact is, in the five or so years I've been developing Catch, and using it in several projects - including a >1m LOC financial system - I'm not sure ever had to even do that. As a rule any resources that should be "clean-up" are already in RAII objects. Even if you look beyond unit tests (I use Catch for integration tests and others) things like file handles, database connections etc are all wrapped in objects that clean themselves up.
In any case we're getting off the original topic. @rcdailey are you happy that your original question has been answered (by @jfemia, augmented by the link I posted)? Can I close the issue?
I'll close it for you. Yes I feel satisfied in the answer. The main motivation for asking was because of some global singletons we have in our legacy code base. Those have to be initialized and released. But I can do that with a simple wrapper.
I'm going to reopen this because I've hit another scenario that RAII doesn't solve easily for me. I'll explain.
My tests are structured in two ways, depending on the test translation unit. A series of sibling TEST_CASE macros, or TEST_CASE with several SECTIONs inside.
Our code base unfortunately is legacy and has a lot of singletons in it. Most singletons are used behind the scenes, so you don't even really realize when you're sharing global state between test cases and/or sections. Furthermore, many of these singletons leave behind files in working directories that cause state sharing between test cases.
To help automate this cleanup, after every TEST_CASE completes (and I expect 1 TEST_CASE to complete for every SECTION within it that is executed), I delete a few common global singletons and delete some files from the test working directory. This guarantees that each test case or section starts "fresh". I have hundreds of test cases, so adding boilerplate RAII objects to the top of each one to do this cleanup is not sustainable and not feasible.
I need a way to tie into test case & section scope to do this. So far I've leveraged a custom reporter to do this, but when you have SECTIONs nested in TEST_CASE, you don't get a testCaseEnded() callback. So I can't do this reliably.
What other options do I have? I'm fine using a reporter for this (even though this feels like I'm abusing an unrelated system) but we would need to require that every sectionEnded() be followed by a testCaseEnded().
@philsquared This is a step in the right direction, but it doesn't address the semantic issue. Let's say I wanted to perform my global state cleanup at the start of every test case, then:
When I have a test case with 2 sections, testCaseStarting() is not invoked between each SECTION. I expect this to happen because the order of execution through the nested scopes is:
TEST_CASE("1")->SECTION("1")TEST_CASE("1")->SECTION("2")
However, because my test case does some setup prior to each section (shared logic between the sections), I can't execute the global state cleanup at the start of each section. I see there's a testGroupStarting(), but this doesn't appear to be override-able from TestEventListenerBase. What would your recommendation be? I need to be able to execute global cleanup state just 1 time per single test case run, regardless of now many nested sections there are.
I've created a pull request that addresses the inconsistency with startingTestCase() with regards to sections. Please let me know if this is an acceptable change upstream. This makes the behavior a lot more consistent and allows me to do cleanup properly no matter how many nested leaf sections there are in a test case.
Sorry I misreaded you (you mentioned "before any test cases") I leave my old answer here:
There's no need for setup teardown:
TEST_CASTE(){ // setup SECTION(){ } // TEARDOWN }the setup and teardown are implicit in how catch work (wich is what I like most). Of course the same apply to nested cases:
TEST_CASTE(){ // setup A SECTION(){ // A // setup B,C SECTION(){ //B } SECTION(){ // C } // TEARDOWN B,C } // TEARDOWN A }
It would be awesome if this example worked. Is it possible to implement some new assertion macros that would stop the current section from executing, but not the rest of the code? Like the "break" keyword.
I've got a similar problem with ZMQ. For example:
zmqpp::context context;
ObjectA instanceA(&context); //instanceA will use context to create threads to send and receive data from ZMQ
...
...
REQUIRE(false);
When the test case fails, the objects are destroyed in the reverse order they are created. However, instanceA's destructor hangs on a receive thread join() as that thread is a blocking ZMQ receive call. ZMQ is designed to unblock receive calls if the context is destroyed, but that can't happen because the ZMQ context is declared before instanceA.
I've got a similar problem with ZMQ. For example:
zmqpp::context context; ObjectA instanceA(&context); //instanceA will use context to create threads to send and receive data from ZMQ ... ... REQUIRE(false);When the test case fails, the objects are destroyed in the reverse order they are created. However, instanceA's destructor hangs on a receive thread
join()as that thread is a blocking ZMQ receive call. ZMQ is designed to unblock receive calls if the context is destroyed, but that can't happen because the ZMQ context is declared before instanceA.
I ended up creating a RAII object that is instantiated after the creation of instanceA, hence it will be destroyed first and in its destructor call the ZMQ context destructor in a separate thread.
A RAII object is probably the most general solution (and I strongly recommend having a scope_guard variant in your codebase).
However, if you can move the context and instance objects outside of the TEST_CASE, your problem is will server by the event listeners -- you can destroy the context when it ends. This does not work in cases where you do need the instance to be part of the TEST_CASE, because the event can only happens after the TEST_CASE is exited.
A RAII object is probably the most general solution (and I strongly recommend having a
scope_guardvariant in your codebase).However, if you can move the context and instance objects outside of the
TEST_CASE, your problem is will server by the event listeners -- you can destroy the context when it ends. This does not work in cases where you do need the instance to be part of theTEST_CASE, because the event can only happens after theTEST_CASEis exited.
I do need the context and instance objects to be part of the test case. I'm using a RAII object for now but a scope_guard seems quite elegant and generic. Thanks for the idea.
Hi guys, I'm migrating my tests from gtest to Catch. I have an integration test case with many sections which all write to/read from the same file. In gtest, I'll clean them up using the "TearDown" method in a fixture. How do achieve the same in Catch?
+1 to this issue, though I don't have a solution (has anyone looked at @rcdailey's pull request?).
I just hit this issue with a behind-the-scenes singleton that was accumulating data across tests, causing unexpected behavior. The problems did not happen when I ran 1 test at a time, but then they showed up when I ran the whole suite (as you can imagine), so was head-scratching for a while (:
faced the same problem, a way of doing this can be:
TEST_CASE("mytestcase") {
SECTION(""){
cout<<"setup"<<endl;
}
SECTION(""){
cout<<"\tbefore"<<endl;
SECTION("subcase_1"){
cout<<"\t\tsubcase_1"<<endl;
}
SECTION("subcase_2"){
cout<<"\t\tsubcase_2"<<endl;
}
cout<<"\tafter"<<endl;
}
SECTION(""){
cout<<"teardown"<<endl;
}
}
Another use case (let's not argue about the new / delete, please) is when I want to SKIP one scenario (this will result in memory leak):
TEST_CASE("TC")
{
printf("calling new\n");
auto *i = new int[10];
SECTION("S1")
{
SKIP("Coz reasons");
}
SECTION("S2")
{
REQUIRE(nullptr != i);
}
printf("calling delete\n");
delete[] i;
}
