Escaped characters in environment variables are not escaped in requests

[chapa]

Expected Behavior

With the following environment:

{
  "backslash": "\\"
}

And the following request body:

{
  "from variable": "{{ _.backslash }}",
  "from raw text": "\\"
}

The request body that is sent should be:

{
  "from variable": "\\",
  "from raw text": "\\"
}

Actual Behavior

The request body that is actually sent is:

{
  "from variable": "\",
  "from raw text": "\\"
}

Reproduction Steps

1. Go to https://webhook.site and copy your unique URL to clipboard
2. Use this URL in a new request in Insomnia
3. Define this environment:

   {
     "backslash": "\\"
   }
   

4. Define this JSON request body:

   {
     "from variable": "{{ _.backslash }}",
     "from raw text": "\\"
   }
   

5. Send the request
6. Go back to https://webhook.site and observe the raw content of the request you just sent

Is there an existing issue for this?

• [X] I have searched the issue tracker for this problem.

Additional Information

No response

Insomnia Version

2021.7.2

What operating system are you using?

Other Linux

Operating System Version

Arch Linux

Installation method

insomnia-bin package from AUR

Last Known Working Insomnia version

No response

[filfreire]

Hi @chapa, thanks for reporting this one!

We're also able to reproduce this:

Feel free to report any other issues you come across in the meantime. We'll update you here if there's any changes.

[chapa]

Hi @filfreire, you're welcome!

I took some time to dig into the code to see if I was able to fix the bug, but I realized this is not as simple as I thought.

Environments are defined in JSON format, so when we set { "backslash": "\\" } the intrinsic value of the backslash variable is really just \ (because environment is written in JSON so we need to escape the \ we want to "store as a variable"). The thing is that variables are replaced by their value as is (that's why we have to wrap them inside quotes), so it's normal that "test": "{{ _.backslash }}" is replaced by "test": "\". If we want to have it replaced by "test": "\\" we must set environment to { "backslash": "\\\\" } (intrinsic value \\, the escaped backslash we want to write in our request body).

The problem is that we can't have a variable with a unescaped intrinsic value and use it in a request body where it should be escaped. In this particular case we want to write JSON so \ should be escaped, but it would not be a problem if we were writing in another format like YAML for example (test: \ is a valid YAML). On the other hand, having the environment { "colon": ":" } would be a problem in YAML (test: : is not valid) and not in JSON ({"test": ":"} is valid).

Here are some thoughts on how to solve the problem:

1. Do nothing and let the users escape special chars

Pros:

• Easy to implement (lol)

Cons:

• Users have to know that the intrinsic value of the variable must include escaped chars, and have to know for which format they must escape it
• A variable with special chars can't be used in request body of different formats (special chars are not the same for JSON and YAML, so escaping for JSON can produce a different string in YAML and vice versa)

2. Create template tags

Having several template tags for each format could allow users to escape variables for the format they want. For example there could be a json template tag which could be used like that : {% json(_.variable) %} and that would behave like JSON.stringify. It can be useful in request body to escape strings (or insert non-scalar values 💪), but also in the address bar to url-encode variables (replace special chars by their percent-encoded variant).

Pros:

• Not so hard to implement (I guess)
• Easy to use
• Users can choose which variable they want to escape
• This could also allow the use of non-scalar variables, instead of converting objects to [object Object] and arrays to comma-separated lists

Cons:

• Users still need to know that values are not escaped and they have to use template tags
• It could be boring to use template tags when there are a lot of variables (maybe a marginal case)

3. Automatically escape every variable depending on the context (the format in which they are used)

Choosing "JSON" as request body format would have the same effect as if all variables used in the request body were wrapped by the json template tag mentioned in point 2. Also, all variables used in the address bar would be as if they were wrapped by the url template tag.

Pros:

• Users have nothing to do and nothing to think about, it just works
• Like in point 2, allow the use of non-scalar variables (without nothing more to do or think about than just using the variable)

Cons:

• Maybe harder to implement
• (⚠ Big one) It would be a breaking change because users would not have to wrap variables into quotes as they do now
• All variables are escaped and there is no opt-out, there may be other situations I haven't thought of where it could be a problem

I may have gone too far in my thinking 😅 but I hope it will give you some ideas 😉

[kdeenanauth]

For folks coming here trying to escape encoded JSON from environment variables and having issues escaping with quotations, I found that the search replace plugin (https://insomnia.rest/plugins/insomnia-plugin-stringreplace) perfect for replacing single quotes with double quotes.

Given:

{
  "manifestJsonObj": "{'id': '{{ _.manifestId }}', 'files': [{'id': '{{ _.fileId }}', 'name': '{{ _.filename }}', 'filename': '{{ _.fullFilename }}' } ] }"
}

In a request:

{
  "id": "test123-file-dd2a498b11154663a20faf9cf28e6d1bz",
  "manifest": "{% replace 'regexp', '/\'/g', '\\"', '', _.manifestJsonObj %}"
}

Results with:

{
   "id": "test123-file-dd2a498b11154663a20faf9cf28e6d1bz",
   "manifest": "{\"id\": \"test123-manifest-2d89ce5c5b0b4e739fb5f1290cb7cc8cz\", \"files\": [{\"id\": \"test123-file-dd2a498b11154663a20faf9cf28e6d1bz\", \"name\": \"profileimage\", \"filename\": \"profileimages__childImage_8919372.jpg\" } ] }"
}