CT hook cth_surefire enabled hides reason for init_per_suite crash

[tsloughter]

This may be a bug in cth_surefire in OTP and not with rebar3 but it needs investigation.

If the hook is configured to be used:

{ct_opts, [{ct_hooks, [cth_surefire]}]}

And a crash happens in init_per_suite of a suite the logs you get from rebar3 are just:

"cth_surefire:pre_init_per_suite/3 CTH call failed"

Removing the hook results in the actual cause of the crash to be printed by rebar3.

[tsloughter]

So another wrinkle.

It will continue to fail, even if the suite's init_per_suite is fixed, until _build/test is deleted! So while there still may be something about the initial failure and rebar3 not giving the actual init_per_suite failure, it is definitely an issue with whatever state cth_surefire loads from previous runs as well.

[ferd]

Yeah that seems to point to the hook itself having a bug. That kind of error message is what I'd get in cth_readable whenever I messed up some log handling and things would blow up.

[paulo-ferraz-oliveira]

On rebar 3.14.1 it returns

init_per_suite: FAILED
%%% ams_SUITE ==> {thrown,a}

I find no reference to CTH in the logs. It's either fixed or I'm doing something wrong to try and replicate it.

[ferd]

I this is not something we can fix, just something we can track as it has to do with how CT hooks themselves handle their lifecycle.

[Falsen]

I think I am having the same issue, any progress on the issue?

[ferd]

Nope. This issue, last time I saw it, came from enabling the maybe expressions in the compiler but not in the VM (need to set ERL_FLAGS="-enable-feature all"), and random modules not being loadable by common test. It's something rather outside of our control.