Elixir conversion doesn't work for POST Requests with nested JSON params

[Leo328]

Hey there,

Thanks in advance for taking a look at this.

I'm trying to convert this curl request into Elixir and it doesn't seem to work correctly. To the contrary, it does convert into Python correctly.

Here is the Curl:

curl 'https://www.rockauto.com/catalog/catalogapi.php' \
  -H 'Connection: keep-alive' \
  -H 'Accept: text/plain, */*; q=0.01' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Origin: https://www.rockauto.com' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://www.rockauto.com/' \
  -H 'Accept-Language: en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7' \
  -H 'Cookie: idlist=0; mkt_EU=false; mkt_CA=true; mkt_MX=true; year_2000=true; viewcurrency=AUD; mkt_US=true; ck=1; saved_server=eyJ0cyI6IjIwMjAtMDYtMjEgMjM6NTQ6MjciLCJuYW1lIjoid3d3Mi5yb2NrYXV0by5jb20iLCJkbnMiOiJ3d3cyLnJvY2thdXRvLmNvbSJ9' \
  --data-raw 'func=navnode_fetch&payload=%7B%22jsn%22%3A%7B%22groupindex%22%3A%2221%22%2C%22nodetype%22%3A%22make%22%2C%22tab%22%3A%22catalog%22%2C%22idepth%22%3A0%2C%22make%22%3A%22AUDI%22%2C%22parentgroupindex%22%3A%22PNODE__GIP%22%2C%22jsdata%22%3A%7B%22markets%22%3A%5B%7B%22c%22%3A%22US%22%2C%22y%22%3A%22Y%22%2C%22i%22%3A%22Y%22%7D%2C%7B%22c%22%3A%22CA%22%2C%22y%22%3A%22Y%22%2C%22i%22%3A%22Y%22%7D%2C%7B%22c%22%3A%22MX%22%2C%22y%22%3A%22Y%22%2C%22i%22%3A%22Y%22%7D%5D%2C%22mktlist%22%3A%22US%2CCA%2CMX%22%2C%22showForMarkets%22%3A%7B%22US%22%3Atrue%2C%22CA%22%3Atrue%2C%22MX%22%3Atrue%7D%2C%22importanceByMarket%22%3A%7B%22US%22%3A%22Y%22%2C%22CA%22%3A%22Y%22%2C%22MX%22%3A%22Y%22%7D%2C%22Show%22%3A1%7D%2C%22label%22%3A%22AUDI%22%2C%22href%22%3A%22https%3A%2F%2Fwww.rockauto.com%2Fen%2Fcatalog%2Faudi%22%2C%22labelset%22%3Atrue%2C%22ok_to_expand_single_child_node%22%3Atrue%2C%22bring_listings_into_view%22%3Atrue%2C%22expand_after_load%22%3Atrue%2C%22fetching%22%3Atrue%7D%2C%22max_group_index%22%3A253%7D&api_json_request=1&sctchecked=1&scbeenloaded=false&curCartGroupID=' \
  --compressed

Here is the output in Elixir:

request = %HTTPoison.Request{
    method: :post,
    url: "https://www.rockauto.com/catalog/catalogapi.php",
    options: [hackney: [cookies: [~s|idlist=0; mkt_EU=false; mkt_CA=true; mkt_MX=true; year_2000=true; viewcurrency=AUD; mkt_US=true; ck=1; saved_server=eyJ0cyI6IjIwMjAtMDYtMjEgMjM6NTQ6MjciLCJuYW1lIjoid3d3Mi5yb2NrYXV0by5jb20iLCJkbnMiOiJ3d3cyLnJvY2thdXRvLmNvbSJ9|]]],
    headers: [
      {~s|Connection|, ~s|keep-alive|},
      {~s|Accept|, ~s|text/plain, */*; q=0.01|},
      {~s|X-Requested-With|, ~s|XMLHttpRequest|},
      {~s|User-Agent|, ~s|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36|},
      {~s|Content-Type|, ~s|application/x-www-form-urlencoded|},
      {~s|Origin|, ~s|https://www.rockauto.com|},
      {~s|Sec-Fetch-Site|, ~s|same-origin|},
      {~s|Sec-Fetch-Mode|, ~s|cors|},
      {~s|Sec-Fetch-Dest|, ~s|empty|},
      {~s|Referer|, ~s|https://www.rockauto.com/|},
      {~s|Accept-Language|, ~s|en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7|},
    ],
    params: [],
    body: [
      {~s|func|, ~s|navnode_fetch|},
      {~s|payload|, ~s|{"jsn":{"groupindex":"21","nodetype":"make","tab":"catalog","idepth":0,"make":"AUDI","parentgroupindex":"PNODE__GIP","jsdata":{"markets":[{"c":"US","y":"Y","i":"Y"},{"c":"CA","y":"Y","i":"Y"},{"c":"MX","y":"Y","i":"Y"}],"mktlist":"US,CA,MX","showForMarkets":{"US":true,"CA":true,"MX":true},"importanceByMarket":{"US":"Y","CA":"Y","MX":"Y"},"Show":1},"label":"AUDI","href":"https://www.rockauto.com/en/catalog/audi","labelset":true,"ok_to_expand_single_child_node":true,"bring_listings_into_view":true,"expand_after_load":true,"fetching":true},"max_group_index":253}|},
      {~s|api_json_request|, ~s|1|},
      {~s|sctchecked|, ~s|1|},
      {~s|scbeenloaded|, ~s|false|},
      {~s|curCartGroupID|, ""}
    ]
  }

  response = HTTPoison.request(request)

When running this code, I get the following error:

response = HTTPoison.request(request)
** (ArgumentError) argument error
    :erlang.iolist_to_binary([{"func", "navnode_fetch"}, {"payload", "{\"jsn\":{\"groupindex\":\"21\",\"nodetype\":\"make\",\"tab\":\"catalog\",\"idepth\":0,\"make\":\"AUDI\",\"parentgroupindex\":\"PNODE__GIP\",\"jsdata\":{\"markets\":[{\"c\":\"US\",\"y\":\"Y\",\"i\":\"Y\"},{\"c\":\"CA\",\"y\":\"Y\",\"i\":\"Y\"},{\"c\":\"MX\",\"y\":\"Y\",\"i\":\"Y\"}],\"mktlist\":\"US,CA,MX\",\"showForMarkets\":{\"US\":true,\"CA\":true,\"MX\":true},\"importanceByMarket\":{\"US\":\"Y\",\"CA\":\"Y\",\"MX\":\"Y\"},\"Show\":1},\"label\":\"AUDI\",\"href\":\"https://www.rockauto.com/en/catalog/audi\",\"labelset\":true,\"ok_to_expand_single_child_node\":true,\"bring_listings_into_view\":true,\"expand_after_load\":true,\"fetching\":true},\"max_group_index\":253}"}, {"api_json_request", "1"}, {"sctchecked", "1"}, {"scbeenloaded", "false"}, {"curCartGroupID", ""}])
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney_request.erl:352: :hackney_request.handle_body/4
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney_request.erl:87: :hackney_request.perform/2
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney.erl:376: :hackney.send_request/2
    (httpoison 1.5.1) lib/httpoison/base.ex:786: HTTPoison.Base.request/6

For comparison, here is the Python returned that works as intended:

import requests

  cookies = {
      'idlist': '0',
      'mkt_EU': 'false',
      'mkt_CA': 'true',
      'mkt_MX': 'true',
      'year_2000': 'true',
      'viewcurrency': 'AUD',
      'mkt_US': 'true',
      'ck': '1',
      'saved_server': 'eyJ0cyI6IjIwMjAtMDYtMjEgMjM6NTQ6MjciLCJuYW1lIjoid3d3Mi5yb2NrYXV0by5jb20iLCJkbnMiOiJ3d3cyLnJvY2thdXRvLmNvbSJ9',
  }

  headers = {
      'Connection': 'keep-alive',
      'Accept': 'text/plain, */*; q=0.01',
      'X-Requested-With': 'XMLHttpRequest',
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36',
      'Content-Type': 'application/x-www-form-urlencoded',
      'Origin': 'https://www.rockauto.com',
      'Sec-Fetch-Site': 'same-origin',
      'Sec-Fetch-Mode': 'cors',
      'Sec-Fetch-Dest': 'empty',
      'Referer': 'https://www.rockauto.com/',
      'Accept-Language': 'en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7',
  }

  data = {
    'func': 'navnode_fetch',
    'payload': '{"jsn":{"groupindex":"21","nodetype":"make","tab":"catalog","idepth":0,"make":"AUDI","parentgroupindex":"PNODE__GIP","jsdata":{"markets":[{"c":"US","y":"Y","i":"Y"},{"c":"CA","y":"Y","i":"Y"},{"c":"MX","y":"Y","i":"Y"}],"mktlist":"US,CA,MX","showForMarkets":{"US":true,"CA":true,"MX":true},"importanceByMarket":{"US":"Y","CA":"Y","MX":"Y"},"Show":1},"label":"AUDI","href":"https://www.rockauto.com/en/catalog/audi","labelset":true,"ok_to_expand_single_child_node":true,"bring_listings_into_view":true,"expand_after_load":true,"fetching":true},"max_group_index":253}',
    'api_json_request': '1',
    'sctchecked': '1',
    'scbeenloaded': 'false',
    'curCartGroupID': ''
  }

  response = requests.post('https://www.rockauto.com/catalog/catalogapi.php', headers=headers, cookies=cookies, data=data)

Thank You.

[ah0y]

the reason for this is because when content type is application/x-www-form-urlencoded the body needs to be uri encoded. take a look at https://elixirforum.com/t/help-with-httpoison-post/11315/4 for more info

[verhovsky]

This was fixed in #453

The issue is that instead of

    body: [
      {~s|func|, ~s|navnode_fetch|},
      {~s|payload|, ~s|{"jsn":{"groupindex":"21","nodetype":"make","tab":"catalog","idepth":0,"make":"AUDI","parentgroupindex":"PNODE__GIP","jsdata":{"markets":[{"c":"US","y":"Y","i":"Y"},{"c":"CA","y":"Y","i":"Y"},{"c":"MX","y":"Y","i":"Y"}],"mktlist":"US,CA,MX","showForMarkets":{"US":true,"CA":true,"MX":true},"importanceByMarket":{"US":"Y","CA":"Y","MX":"Y"},"Show":1},"label":"AUDI","href":"https://www.rockauto.com/en/catalog/audi","labelset":true,"ok_to_expand_single_child_node":true,"bring_listings_into_view":true,"expand_after_load":true,"fetching":true},"max_group_index":253}|},
      {~s|api_json_request|, ~s|1|},
      {~s|sctchecked|, ~s|1|},
      {~s|scbeenloaded|, ~s|false|},
      {~s|curCartGroupID|, ""}
    ]

it needs to be

    body: {:form, [
      {~s|func|, ~s|navnode_fetch|},
      {~s|payload|, ~s|{"jsn":{"groupindex":"21","nodetype":"make","tab":"catalog","idepth":0,"make":"AUDI","parentgroupindex":"PNODE__GIP","jsdata":{"markets":[{"c":"US","y":"Y","i":"Y"},{"c":"CA","y":"Y","i":"Y"},{"c":"MX","y":"Y","i":"Y"}],"mktlist":"US,CA,MX","showForMarkets":{"US":true,"CA":true,"MX":true},"importanceByMarket":{"US":"Y","CA":"Y","MX":"Y"},"Show":1},"label":"AUDI","href":"https://www.rockauto.com/en/catalog/audi","labelset":true,"ok_to_expand_single_child_node":true,"bring_listings_into_view":true,"expand_after_load":true,"fetching":true},"max_group_index":253}|},
      {~s|api_json_request|, ~s|1|},
      {~s|sctchecked|, ~s|1|},
      {~s|scbeenloaded|, ~s|false|},
      {~s|curCartGroupID|, ""}
    ]}