Incorrect group names when groups running in parallel

[michalmuskala]

trafficstars

Given the following suite foo_SUITE:

-module(foo_SUITE).

-export([
    suite/0,
    all/0,
    groups/0,
    init_per_suite/1,
    end_per_suite/1,
    init_per_group/2,
    end_per_group/2,
    init_per_testcase/2,
    end_per_testcase/2
]).

-export([a/1, b/1, c/1]).

suite() ->
    [{timetrap, {seconds, 10}}].

init_per_suite(Config) ->
    Config.

end_per_suite(_Config) ->
    ok.

init_per_group(_GroupName, Config) ->
    Config.

end_per_group(_GroupName, _Config) ->
    ok.

init_per_testcase(_TestCase, Config) ->
    Config.

end_per_testcase(_TestCase, _Config) ->
    ok.

groups() ->
    [
        {as, [parallel], [
            a,
            {group, bs}
        ]},
        {bs, [parallel], [
            b
        ]},
        {all, [parallel], [
            {group, as},
            c
        ]}
    ].

all() ->
    [
        {group, all}
    ].

a(_) -> timer:sleep(1000), error(a).

b(_) -> timer:sleep(1000), error(b).

c(_) -> timer:sleep(1000), error(c).

Running the default rebar3 ct produces:

%%% foo_SUITE:
%%% foo_SUITE ==> all.as.bs.a: FAILED
%%% foo_SUITE ==> {a,[{foo_SUITE,a,1,
               [{file,"/data/users/micmus/github/erlfmt/test/foo_SUITE.erl"},
                {line,58}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1748}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1263}]},
    {test_server,run_test_case_eval,9,
                 [{file,"test_server.erl"},{line,1195}]}]}

%%% foo_SUITE ==> all.as.bs.b: FAILED
%%% foo_SUITE ==> {b,[{foo_SUITE,b,1,
               [{file,"/data/users/micmus/github/erlfmt/test/foo_SUITE.erl"},
                {line,60}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1748}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1263}]},
    {test_server,run_test_case_eval,9,
                 [{file,"test_server.erl"},{line,1195}]}]}

%%% foo_SUITE ==> all.c: FAILED
%%% foo_SUITE ==> {c,[{foo_SUITE,c,1,
               [{file,"/data/users/micmus/github/erlfmt/test/foo_SUITE.erl"},
                {line,62}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1748}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1263}]},
    {test_server,run_test_case_eval,9,
                 [{file,"test_server.erl"},{line,1195}]}]}

Note the incorrecy name all.as.bs.a - test a does not belong to bs group.

[ferd]

I believe there is no way to fix this. The CT hooks just have disjoint calls for init_per_<whatever>, and test cases themselves carry no information about the groups they belong to; this library reconstructs them by observing the order. Running various groups in parallel therefore results in the naming being more or less randomly assigned to the test cases since the CT hook's scope is globally shared.

A workaround could be to try and modify the Config value returned by the hook (see docs) but this in turn means that the CT hooks start modifying test data on behalf of test cases by injecting arbitrary configuration values that are then readable within each test case's config.

I'm not sure, in this case, whether the fix is worse than the problem.

[TheGeorge]

I have been struggling with this as well. Without changes to CT I think there is no way (even with writing information written to Config) to reliable fix this. There are two ways of carrying data in hooks:

• the hook state, which is shared, this does not work reliable for parallel groups
• Config, which get's carried over correctly, but then is not available in the on_tc_* callbacks (otherwise, one could use a very special key for the location data)