[Bug] JSON full body assertion fails when inline but succeeds when externalised

[jpluscplusm]

Heya 👋

Full body assertions fail against the same JSON source when it's presented inline in a .hurl file, versus succeeding when asserted against in a separate, external file.

In this example, I:

• use hurl to fetch a well-known static JSON document and store it in a file
• duplicate the initial .hurl file and add a content assertion against the file
• duplicate the initial .hurl file and add a content assertion against the file's content, inlined
• show that the new externalised-content and inlined-content assertion .hurl files succeed and fail respectively, despite being derived from the same source and presented appropriately, as per the docs
• modify the file's contents and show that the now-obviously-incorrect externalised-content assertion fails, as expected.
• show the incorrectly-failing .hurl file, for clarity

Something seems off, here. My reading of the docs (at https://hurl.dev/docs/asserting-response.html#body) makes me think I should be able to simply place the HTTP JSON body response in the .hurl file, as per my example:

If the body of the response is a JSON string or a XML string, the body assertion can be directly inserted without any modification

$ cat no-assertions.hurl 
GET https://httpbin.org/json

HTTP/* 200
$ hurl no-assertions.hurl > content.json
$ cp no-assertions.hurl assert-file-contents.hurl
$ echo 'file,content.json;' >> assert-file-contents.hurl 
$ cp no-assertions.hurl assert-inline.hurl
$ cat content.json >> assert-inline.hurl 
$ hurl --test assert-file-contents.hurl 
assert-file-contents.hurl: running [1/1]
assert-file-contents.hurl: success
--------------------------------------------------------------------------------
executed: 1
success: 1 (100.0%)
failed: 0 (0.0%)
execution time: 549ms
$ hurl --test assert-inline.hurl 
assert-inline.hurl: running [1/1]
error: Assert Body Value
  --> assert-inline.hurl:4:1
   |
 4 | {
   | ^ actual value is <{
  "slideshow": {
    "author": "Yours Truly", 
    "date": "date of publication", 
    "slides": [
      {
        "title": "Wake up to WonderWidgets!", 
        "type": "all"
      }, 
      {
        "items": [
          "Why <em>WonderWidgets</em> are great", 
          "Who <em>buys</em> WonderWidgets"
        ], 
        "title": "Overview", 
        "type": "all"
      }
    ], 
    "title": "Sample Slide Show"
  }
}
>
   |

assert-inline.hurl: failure
--------------------------------------------------------------------------------
executed: 1
success: 0 (0.0%)
failed: 1 (100.0%)
execution time: 864ms
$ echo '{ "fail": true }' > content.json 
$ hurl --test assert-file-contents.hurl 
assert-file-contents.hurl: running [1/1]
error: Assert Body Value
  --> assert-file-contents.hurl:4:1
   |
 4 | file,content.json;
   | ^ actual value is <hex, 7b0a202022736c69646573686f77223a207b0a2020202022617574686f72223a2022596f757273205472756c79222c200a202020202264617465223a202264617465206f66207075626c69636174696f6e222c200a2020202022736c69646573223a205b0a2020202020207b0a2020202020202020227469746c65223a202257616b6520757020746f20576f6e6465725769646765747321222c200a20202020202020202274797065223a2022616c6c220a2020202020207d2c200a2020202020207b0a2020202020202020226974656d73223a205b0a2020202020202020202022576879203c656d3e576f6e646572576964676574733c2f656d3e20617265206772656174222c200a202020202020202020202257686f203c656d3e627579733c2f656d3e20576f6e64657257696467657473220a20202020202020205d2c200a2020202020202020227469746c65223a20224f76657276696577222c200a20202020202020202274797065223a2022616c6c220a2020202020207d0a202020205d2c200a20202020227469746c65223a202253616d706c6520536c6964652053686f77220a20207d0a7d0a;>
   |

assert-file-contents.hurl: failure
--------------------------------------------------------------------------------
executed: 1
success: 0 (0.0%)
failed: 1 (100.0%)
execution time: 775ms
$ cat assert-inline.hurl 
GET https://httpbin.org/json

HTTP/* 200
{
  "slideshow": {
    "author": "Yours Truly", 
    "date": "date of publication", 
    "slides": [
      {
        "title": "Wake up to WonderWidgets!", 
        "type": "all"
      }, 
      {
        "items": [
          "Why <em>WonderWidgets</em> are great", 
          "Who <em>buys</em> WonderWidgets"
        ], 
        "title": "Overview", 
        "type": "all"
      }
    ], 
    "title": "Sample Slide Show"
  }
}

[fabricereix]

Thanks a lot for your detailed steps.

You're right, contrary to what the doc says, you can not always take the JSON response to your Hurl file.

There is no ambiguities ith the external file. Hurl makes a binary (byte by byte) comparison between actual and expected response.

The inline JSON within Hurl is parsed without any trailing space. This is inherent to the Hurl File format. The response from https://httpbin.org/json contains a trailing newline.

for example, the following Hurl file has all the same expected JSON { "name": "Bob" }.

test.hurl
GET http://example.com/json
HTTP/* 200
{ "name": "Bob" }

GET http://example.com/json
HTTP/* 200
{ "name": "Bob" }  # not part of the expected JSON

This can be checked with JSON export of hurlfmt.

$ hurlfmt --format json /tmp/test.hurl| jq
{
  "entries": [
    {
      "request": {
        "method": "GET",
        "url": "http://example.com/json"
      },
      "response": {
        "status": 200,
        "body": {
          "type": "json",
          "value": {
            "name": "Bob"
          }
        }
      }
    },
    {
      "request": {
        "method": "GET",
        "url": "http://example.com/json"
      },
      "response": {
        "status": 200,
        "body": {
          "type": "json",
          "value": {
            "name": "Bob"
          }
        }
      }
    }
  ]
}

In order to test a JSON with a trailing space, you could use raw string. But for sure, this makes using the JSON body more limited and hard to understand in case of error.

We are thinking to change the current textual comparison for JSON to a "semantic" JSON comparison, thus ignoring whitespace in the JSON response.

[ad-si]

@fabricereix A semantic JSON comparison would be awesome! 👍

[marcodpt]

I really needed semantic JSON comparison implemented to permanently migrate my tests to this wonderful project!