xprof icon indicating copy to clipboard operation
xprof copied to clipboard

Make occasionally failing ct testcases more robust

Open gomoripeti opened this issue 7 years ago • 0 comments

Some of the ct testcases fail from time to time but not deterministically

This failure is quite common and hopeless to fix because of how timers work in Erlang. (A call to timer:sleep(20) is guaranteed to last at least 20 ms but there is no upper limit)

%%% xprof_tracing_SUITE ==> long_call: FAILED
%%% xprof_tracing_SUITE ==> 
Failure/Error: ?assertMatch({ true , _ }, { Min < 22 * 1000 , Min })
  expected: = { true , _ }
       got: {false,24480}      line: 396

Another example when samples are not yet available. (This happens quite often in long_call testcase)

%%% xprof_http_e2e_SUITE ==> capture_data_when_traced_test: FAILED
%%% xprof_http_e2e_SUITE ==>
Failure/Error: ?assertEqual(1, length ( proplists : get_value ( << "items" >> , Data ) ))
expected: 1
got: 0 line: 210

To make snapshots more deterministic (without waiting 1-2 seconds) is to add a function to xprof_core_trace_handler to trigger taking one synchronously (and callerlang:trace_delivered before). But as the key of the snapshots is the timestamp in seconds, there can only be one snapshot per second (and some testcases require two).

Or captured data is not yet available:

%%% xprof_http_e2e_SUITE ==> capture_data_with_formatted_exception_test: FAILED
%%% xprof_http_e2e_SUITE ==>
Failure/Error: ?assertMatch([ << "** exception error: no match of right hand side value ok" >> ], [ proplists : get_value ( << "res" >> , Item ) || Item <- proplists : get_value ( << "items" >> , Data ) ])
expected: = [ << "** exception error: no match of right hand side value ok" >> ]
got: [] line: 244

gomoripeti avatar Oct 24 '17 19:10 gomoripeti