Orange-OpenSource
Failing assertions should print the response body
Currently it is very annoying to use hurl to debug why a test failed. Say you have this file:
GET https://example.com/
HTTP 404 # failing assertion
Now, let's say this test fails. You thought example.com should return 404, but it is returning something else. In order to diagnose why, you want to look at the response body. For a JSON API, this is much more important than example.com, but it serves to illustrate the problem.
What is the current bug behavior?
All you get is the failed assertion. If you want to see the response body, you have to comment out the assertion, and run it again. This is inconvenient enough for a normal test; I imagine it's 100x worse for flaky tests in a bigger test suite.
error: Assert status code
--> ./example.hurl:2:6
|
2 | HTTP 201
| ^^^ actual value is <200>
|
What is the expected correct behavior?
Print out the body when a test fails, in much the same way cargo test will print the stdout only for failing test. You can probably leave it unprinted when there are assertions but they all passed. If the response body is really huge, you can probably truncate it, if you mention it's been snipped off in the output.
<!doctype html>
<html><head><title>You have navigated to a real website! It's not 404. That's why the test failed! </title></head>
...
</html>
--> ./example.hurl:2:6
|
2 | HTTP 201
| ^^^ actual value is <200>
|
Execution context
- Hurl Version (
hurl --version): `hurl 4.0.0 libcurl/8.2.1-DEV OpenSSL/3.0.2 zlib/1.2.11 Features (libcurl): alt-svc AsynchDNS HSTS IPv6 libz SSL UnixSockets Features (built-in): brotli - Operating system and version: debian
Hi @cormac-ainc, for now you can use --error-format=long for debugging purpose:
- On success:
echo -e "GET https://google.fr \n HTTP 301" | hurl --test --error-format=long
-: Running [1/1]
-: Success (1 request(s) in 97 ms)
--------------------------------------------------------------------------------
Executed files: 1
Succeeded files: 1 (100.0%)
Failed files: 0 (0.0%)
Duration: 101 ms
- On error:
$ echo -e "GET https://google.fr \n HTTP 404" | hurl --test --error-format=long
-: Running [1/1]
HTTP/2 301
location: https://www.google.fr/
content-type: text/html; charset=UTF-8
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-sKhbxAmDiD0Qdve6Wpl8pA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
date: Tue, 22 Aug 2023 08:26:00 GMT
expires: Tue, 22 Aug 2023 08:26:00 GMT
cache-control: private, max-age=2592000
server: gws
content-length: 219
x-xss-protection: 0
x-frame-options: SAMEORIGIN
set-cookie: CONSENT=PENDING+182; expires=Thu, 21-Aug-2025 08:26:00 GMT; path=/; domain=.google.fr; Secure
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.fr/">[here](https://www.google.fr/)</A>.
</BODY></HTML>
error: Assert status code
--> -:2:7
|
2 | HTTP 404
| ^^^ actual value is <301>
|
-: Failure (1 request(s) in 98 ms)
--------------------------------------------------------------------------------
Executed files: 1
Succeeded files: 0 (0.0%)
Failed files: 1 (100.0%)
Duration: 102 ms
We are thinking about supporting --write-out curl option, take a look at https://github.com/Orange-OpenSource/hurl/issues/1745#issuecomment-1634103514.
