Numbers in Unit XML wrong

[mbeckerle-xqueue]

Describe the bug

I have been been fuzzing for some hours and got a report telling me the following:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="32693" tests="32693" time="0.0">
	<testsuite disabled="0" errors="0" failures="32693" name="API Fuzzer" skipped="0" tests="32693" time="0">
		<testcase name="136" status="failed">

According to this, the time was 0 seconds and all tests failed. However, looking at the last failed record I see:

<testcase name="61525" status="failed">

So I have at least 61525 tests. So the numbers above are not correct. However, the numbers of failures is correct as in many cases the API returned 505, which is not critical but will be fixed, however, I would like to see the correct numbers in the report.

Software environment (please complete the following information):

• OS: Ubuntu 20.04.3 LTS
• Python version: Python 3.8.10
• APIFuzzer Version: 0.9.12

[KissPeter]

Hi, Any PR is welcome, bit v 0.9.13 supports it. Please test it

[mbeckerle-xqueue]

Hi Peter,

thank you for looking into this and adding the numbers so fast. Elapsed time is now fine and evaluating progress over time and overall time works great.

The other thing is still not working as I would expect it to behave: it does not count the number of tests correctly:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="34" tests="34" time="5.8366954401135445">
	<testsuite disabled="0" errors="0" failures="34" name="API Fuzzer" skipped="0" tests="34" time="5.8366954401135445" timestamp="1646639974.3845608">
		[...]
		<testcase name="60" time="0.176842" timestamp="1646639968.669649" status="failed">
			<failure type="failure" message="{&quot;status&quot;: &quot;failed&quot;, &quot;name&quot;: [{&quot;Authorization&quot;: &quot;***redacted***&quot;}], &quot;sub_reports&quot;: [], &quot;test_number&quot;: 60, &quot;state&quot;: &quot;COMPLETED&quot;, &quot;request_url&quot;: &quot;https://***redacted***/contacts/count?updated_after=' or '1'='1&quot;, &quot;request_method&quot;: &quot;GET&quot;, &quot;request_headers&quot;: &quot;{\&quot;User-Agent\&quot;: \&quot;APIFuzzer 0.9.13\&quot;, \&quot;Accept-Encoding\&quot;: \&quot;gzip, deflate\&quot;, \&quot;Accept\&quot;: \&quot;*/*\&quot;, \&quot;Connection\&quot;: \&quot;keep-alive\&quot;, \&quot;Authorization\&quot;: \&quot;***redacted***\&quot;}&quot;, &quot;request_body&quot;: {}, &quot;response&quot;: &quot;&quot;, &quot;parsed_status_code&quot;: 505, &quot;reason&quot;: &quot;failed&quot;}"/>
		</testcase>
		<testcase name="88" time="0.167752" timestamp="1646639973.8767242" status="failed">
			<failure type="failure" message="{&quot;status&quot;: &quot;failed&quot;, &quot;name&quot;: [{&quot;Authorization&quot;: &quot;***redacted***&quot;}], &quot;sub_reports&quot;: [], &quot;test_number&quot;: 88, &quot;state&quot;: &quot;COMPLETED&quot;, &quot;request_url&quot;: &quot;https://***redacted***/contacts/count?updated_after=%u0000&quot;, &quot;request_method&quot;: &quot;GET&quot;, &quot;request_headers&quot;: &quot;{\&quot;User-Agent\&quot;: \&quot;APIFuzzer 0.9.13\&quot;, \&quot;Accept-Encoding\&quot;: \&quot;gzip, deflate\&quot;, \&quot;Accept\&quot;: \&quot;*/*\&quot;, \&quot;Connection\&quot;: \&quot;keep-alive\&quot;, \&quot;Authorization\&quot;: \&quot;***redacted***\&quot;}&quot;, &quot;request_body&quot;: {}, &quot;response&quot;: &quot;&quot;, &quot;parsed_status_code&quot;: 500, &quot;reason&quot;: &quot;failed&quot;}"/>
		</testcase>
	</testsuite>
</testsuites>

It still tells me: tests="34" However, as not all tests failed (see last test case that failed has 88 as test ID) I would expect that the report looks more like: failures="34" name="API Fuzzer" skipped="0" tests="88" (same in testsuites report)

[KissPeter]

Hi, Sorry I missed your message. This would require adding a test even it passed. We had projects where we had millions of cases for a long and complicated API. I think it is better to have smaller amount of test cases listed which requires attention than having lot of them which doesn't require attention furthermore may cause issue in the system which consumes it. An additional config parameter could be introduced for that, however the configuration is starting to be a bit complicated already If you consider this, are you still on the side of generate test for passing cases?

[mbeckerle-xqueue]

I understand your point and I am pretty fine with leaving out failed tests from the error report but my expectation was in fact to have a statistic there how many tests passed, how many failed, etc.

So maybe there is a possibility to remember the total amount without placing them in the data model as results?