templating using Request.Body jsonpath will replace characters with HTML character encodings

[MartynDavis]

Description of the bug

When templating is enabled for JSON request/response, if the request body is:

{ "name": "O'Reilly" }

And the response body is set to:

{ "name": "{{ Request.Body 'jsonpath' '$.name' }}"

The Hoverfly will return:

{ "name": "O@apos;Reilly" }

Steps to reproduce the issue

Import attached Hoverfly simulation file hoverfly-templating-example.json, and set Hoverfly to simulate.

Templated POST response is:

{
  "id": {{ randomIntegerRange 100 199 }},
  "name": "{{ Request.Body 'jsonpath' '$.name' }}",
  "username": "{{ Request.Body 'jsonpath' '$.username' }}",
  "email": "{{ Request.Body 'jsonpath' '$.email' }}",
  "address": {
    "street": "{{ Request.Body 'jsonpath' '$.address.street' }}",
    "suite": "{{ Request.Body 'jsonpath' '$.address.suite' }}",
    "city": "{{ Request.Body 'jsonpath' '$.address.city' }}",
    "zipcode": "{{ Request.Body 'jsonpath' '$.address.zipcode' }}",
    "geo": {
      "lat": "{{ Request.Body 'jsonpath' '$.address.geo.lat' }}",
      "lng": "{{ Request.Body 'jsonpath' '$.address.geo.lng' }}"
    }
  },
  "phone": "{{ Request.Body 'jsonpath' '$.phone' }}",
  "website": "{{ Request.Body 'jsonpath' '$.website' }}",
  "company": {
    "name": "{{ Request.Body 'jsonpath' '$.company.name' }}",
    "catchPhrase": "{{ Request.Body 'jsonpath' '$.company.catchPhrase' }}",
    "bs": "{{ Request.Body 'jsonpath' '$.company.bs' }}"
  }
}

Execute the non-proxied POST:

curl  --request POST --data @post-data.json --header "Content-type: application/json" --header "Accept: application/json" http://jsonplaceholder.typicode.com/users

post-data.json contains the following:

{
  "name": "Stephanie Hackett DVM",
  "username": "kiara.langworth",
  "email": "[email protected]",
  "address": {
    "street": "3027 O'hara Lane",
    "suite": "Penthouse",
    "city": "Johnstonmouth",
    "zipcode": "3341",
    "geo": {
      "lat": "14.511382",
      "lng": "46.748701"
    }
  },
  "phone": "0449-415-927",
  "website": "www.hamilton-ohara.net",
  "company": {
    "name": "O'connell-Tillman",
    "catchPhrase": "Synergized tangible parallelism",
    "bs": "facilitate killer bandwidth"
  }
}

Which will return:

{
  "name": "Stephanie Hackett DVM",
  "username": "kiara.langworth",
  "email": "[email protected]",
  "address": {
    "street": "3027 O'hara Lane",
    "suite": "Penthouse",
    "city": "Johnstonmouth",
    "zipcode": "3341",
    "geo": {
      "lat": "14.511382",
      "lng": "46.748701"
    }
  },
  "phone": "0449-415-927",
  "website": "www.hamilton-ohara.net",
  "company": {
    "name": "O'connell-Tillman",
    "catchPhrase": "Synergized tangible parallelism",
    "bs": "facilitate killer bandwidth"
  },
  "id": 11
}

Execute the proxied POST:

curl  --proxy http://localhost:8500 --request POST --data @post-data.json --header "Content-type: application/json" --header "Accept: application/json" http://jsonplaceholder.typicode.com/users

Which will return:

{
  "id": 192,
  "name": "Stephanie Hackett DVM",
  "username": "kiara.langworth",
  "email": "[email protected]",
  "address": {
    "street": "3027 O'hara Lane",
    "suite": "Penthouse",
    "city": "Johnstonmouth",
    "zipcode": "3341",
    "geo": {
      "lat": "14.511382",
      "lng": "46.748701"
    }
  },
  "phone": "0449-415-927",
  "website": "www.hamilton-ohara.net",
  "company": {
    "name": "O'connell-Tillman",
    "catchPhrase": "Synergized tangible parallelism",
    "bs": "facilitate killer bandwidth"
  }
}

The address.street and company.name values contain @apos; instead of ' (single quote).

Observed result

{{ Request.Body 'jsonpath' '$.address.street' }} returns 3027 O'hara Lane {{ Request.Body 'jsonpath' '$.company.name' }} returns O'connell-Tillman

Expected result

{{ Request.Body 'jsonpath' '$.address.street' }} returns 3027 O'hara Lane {{ Request.Body 'jsonpath' '$.company.name' }} returns O'connell-Tillman

Attached files.zip contains:

$ hoverctl version

+----------+-------------+
| hoverctl | master-3593 |
| hoverfly | v1.3.1      |
+----------+-------------+

• hoverfly-templating-example.json - example simulation file
• post-data.json - curl POST data

files.zip

[tommysitu]

We use this library for templating: https://github.com/aymerick/raymond

According to the doc, if you need to unescaped value, you can triple the mustache {{{ 🧔

if you try this, it should work: { "name": "{{{ Request.Body 'jsonpath' '$.name' }}}"

[MartynDavis]

Thank you @tommysitu for the solution.

Triple moustache it is.

I have attached hoverfly-templating-example-triple-moustache.json which is the original JSON file with the triple moustache solution.

With that said, Hoverfly does needs address the issues with escaping JSON when jsonpath is used within templating. Consider the case where double quotes and/or newlines are used within the request JSON. Using triple moustache in this instance does not address the issue as malformed JSON is returned.

Consider the file post-data-bad.json, which contains:

{
  "name": "Stephanie Hackett DVM",
  "username": "kiara.langworth",
  "email": "[email protected]",
  "address": {
    "street": "3027 O'hara Lane",
    "suite": "\"Penthouse\"",
    "city": "Johnstonmouth",
    "zipcode": "3341",
    "geo": {
      "lat": "14.511382",
      "lng": "46.748701"
    }
  },
  "phone": "0449-415-927",
  "website": "www.hamilton-ohara.net",
  "company": {
    "name": "O'connell-Tillman",
    "catchPhrase": "Synergized tangible parallelism\nEat tangerines",
    "bs": "facilitate killer bandwidth"
  },
  "id": 11
}

The suite property contains double quote and catchPhrase contains newline.

When the Hoverfly simulates the POST response, malformed JSON is returned:

curl  --proxy http://localhost:8500 --request POST --data @post-data-bad.json --header "Content-type: application/json" --header "Accept: application/json" http://jsonplaceholder.typicode.com/users

Data returned is:

{
  "id": 123,
  "name": "Stephanie Hackett DVM",
  "username": "kiara.langworth",
  "email": "[email protected]",
  "address": {
    "street": "3027 O'hara Lane",
    "suite": ""Penthouse"",
    "city": "Johnstonmouth",
    "zipcode": "3341",
    "geo": {
      "lat": "14.511382",
      "lng": "46.748701"
    }
  },
  "phone": "0449-415-927",
  "website": "www.hamilton-ohara.net",
  "company": {
    "name": "O'connell-Tillman",
    "catchPhrase": "Synergized tangible parallelism
Eat tangerines",
    "bs": "facilitate killer bandwidth"
  }
}

The double quote characters within the suite property are not escaped, and the newline within the catchPhrase property is not encoded as '\n'.

hoverfly-templating-example-triple-moustache.zip post-data-bad.zip