Testcase Id Prefix

[simonthum]

Problem to solve

I am scripting hurl to run tests over a number of systems. I want to generate a summary report from that activity.

Proposal

I propose to add a test id prefix option, so different test runs will have different ids.

My wild guess is that if my test ids consisted of a constant prefix + filename, the report merge would lead to a summary report automatically.

Right now I have to put them in different directories and put in extra work for a summary/overview. Identical test ids also are likely to confuse other tools that display test results.

Additional context and resources

For context see https://github.com/Orange-OpenSource/hurl/blob/870e09552f92ea397a0b961d095b2cdec0d86e04/packages/hurl/src/report/junit/testcase.rs#L37

Tasks to complete

• [ ] Add a commandline option for prefix (maybe suffix too)
• [ ] derive test id from prefix and filename in every backend
• [ ] Alternatively, push the test id to the hurl result
• [ ] Ideally, document how to use it to create summary reports

[jcamiel]

Hi @simonthum I struggle to understand your issue 😅!

Could you describe your actual workflow, what you're doing with the JUnit / Hurl test etc... and what could be improved? We need to understand what's missing today and, especially, if we can avoid creating a new command line option? We have a lot of command line option today and really want to create a new one only if there is a string need.

Thanks !

[simonthum]

Hi,

I'll give you a shortened flow:

hurl --variable baseURL=https://host1 TheTest.hurl --test --variable more=stuff --report-junit ju.xml
hurl --variable baseURL=https://anotherHost:8443 .\TheTest.hurl --test --variable more=andmore --report-junit ju.xml

I understand the doc as saying ju.xml would now contain two (joined) test results. But it does not for any of the tests I made, it only contains one with an Id derived from TheTest.hurl.

I would want to vary the test id per host and other criteria, hopefully achieving a single report with all the relevant tests included.

E.g.

hurl --variable baseURL=https://host1 TheTest.hurl --test --variable more=stuff --report-junit ju.xml --report-id-prefix=myHost1_ --report-id-suffix=_smoketest 
hurl --variable baseURL=https://anotherHost:8443 .\TheTest.hurl --test --variable more=andmore --report-junit ju.xml --report-id-prefix=myOtherHost_ --report-id-suffix=_smoketest 

Alternatively, I'd be happy with --report-id=myId as well.

I assume I would end up with a single joined report then, as it works well in other cases. What I care about is an overview report and the ability to associate test results.

[simonthum]

@jcamiel ...or maybe hurl doesn't work that way at all? I read that docs that way and may have tested myself into believing it.

If there is no merge capability across hurl invocations, please just close this request. I'll find some way.

[jcamiel]

Hi @simonthum

Unless it has a bug, Hurl should aggregate across Hurl runs, I've tested it:

$ hurl --test --report-junit ju.xml google.hurl 
$ xmllint --format ju.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite tests="1" errors="0" failures="0">
    <testcase id="google.hurl" name="google.hurl" time="0.125"/>
  </testsuite>
</testsuites>
$
$ hurl --test --report-junit ju.xml apple.hurl
$ xmllint --format ju.xml                     
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite tests="1" errors="0" failures="0">
    <testcase id="google.hurl" name="google.hurl" time="0.125"/>
  </testsuite>
  <testsuite tests="1" errors="0" failures="0">
    <testcase id="apple.hurl" name="apple.hurl" time="0.090"/>
  </testsuite>
</testsuites>

=> two hurlcalls produces a, XML with 1 <testsuites> node and 2 <testsuite> nodes.

Can you check your Hurl version hurl --version and also, in verbose mode (hurl --verbose) you should see at the end of the run a log when writing a JUnit report.

[simonthum]

OK, it seems hurl appends into junit (regardless of the id), but in html not at alll. That seems to be a Bug.

So I will use junit for that overview, but for display I have to hope the id is not used as an Identifier and when things are red I would be struggling to find the host and API that turned red. Good enough for now :)

So to identify the offending test and get to the cause, the ability to have a say over the reported test Id would be great!

[jcamiel]

Hi @simonthum it should work with HTML also:

$ hurl --test --report-html report apple.hurl
$ hurl --test --report-html report google.hurl

And the resulted report:

[simonthum]

I'll try to reproduce this result when I get to it.

[simonthum]

Could it be that html merging does not work when one is using --varibales?

All my tests are fine until I combine html and variables - which is the combination I would prefer.

[jcamiel]

Hi @simonthum could you share the output of:

$ hurl --version

And your exact commands so I can try to see if I reproduce it?

I've tested with variable:

$ hurl --test --variable host1=https://apple.com --report-html report apple.hurl
apple.hurl: Success (1 request(s) in 121 ms)
--------------------------------------------------------------------------------
Executed files:    1
Executed requests: 1 (8.1/s)
Succeeded files:   1 (100.0%)
Failed files:      0 (0.0%)
Duration:          124 ms

$ hurl --test --variable host2=https://google.com --report-html report google.hurl
google.hurl: Success (1 request(s) in 126 ms)
--------------------------------------------------------------------------------
Executed files:    1
Executed requests: 1 (7.8/s)
Succeeded files:   1 (100.0%)
Failed files:      0 (0.0%)
Duration:          129 ms

And the Hurl file are:

$ cat apple.hurl
GET {{host1}}

$ cat google.hurl
GET {{host2}}

[simonthum]

I can, but what I want to get working is your result with only "test.hurl". Can you try that?

I want to avoid identical test files for n hosts - yes I could script that, but that's arkward.

[jcamiel]

Hi @simonthum

With this file test.hurl:

HEAD https://{{host}}

Running two tests:

$ hurl --test --report-html report --variable host=apple.com test.hurl
test.hurl: Success (1 request(s) in 72 ms)
--------------------------------------------------------------------------------
Executed files:    1
Executed requests: 1 (13.7/s)
Succeeded files:   1 (100.0%)
Failed files:      0 (0.0%)
Duration:          73 ms
$ hurl --test --report-html report --variable host=google.com test.hurl
test.hurl: Success (1 request(s) in 92 ms)
--------------------------------------------------------------------------------
Executed files:    1
Executed requests: 1 (10.8/s)
Succeeded files:   1 (100.0%)
Failed files:      0 (0.0%)
Duration:          93 ms

Then:

Once again, check (and report ) hurl --version, previous version (< 6.0.0) might differ and might have a bug!

[simonthum]

I use 6.0.0, but I will try again to reproduce your result and report back.

The feature I want is to "see" which line is what in that report. Can you make a suggestion to that effect?