HashConversions does not handle objects with varying keys in arrays

[twelve17]

trafficstars

Using a variant of the example in HashConversions, given object obj:

{
  "name": "Bob",
  "phones": [
    {
      "type": "home" // He finally cut the cord!
    },
    {
      "type": "work",
      "phone": "333-333-3333"
    }
  ]
}

The following params are produced by to_params:

[51] pry> params = HTTParty::HashConversions.to_params(obj)
=> "name=Bob&phones%5B%5D%5Btype%5D=home&phones%5B%5D%5Btype%5D=work&phones%5B%5D%5Bphone%5D=333-333-3333"
[52] pry>)> puts JSON.pretty_generate(CGI.parse(params))
{
  "name": [
    "Bob"
  ],
  "phones[][type]": [
    "home",
    "work"
  ],
  "phones[][phone]": [
    "333-333-3333"
  ]
}

Bob's work phone number is now his home number.

I realize working around this would require inspecting all array entries and compiling a superset of all keys.

Perhaps the bigger issue I'm seeing is that the default behavior of HTTParty.post when body is an object is to delegate to non-standard serialization of the object, versus assuming the caller wants to send JSON (and thus switch to Content-type: application/json).

Since serialization of array query parameters do not seem to have an agreed-to standard, it would seem more sensical to have this be an opt-in code path, and the default be the JSON code path. Especially when you consider the ambiguous format: :json parameter:

HTTParty.post(
  "/some/url",
  format: :json,
  body: {
    "name: "Bob",
    "phones": [
      {
        "type": "home"
      },
      {
        "type": "work",
        "phone": "333-333-3333"
      }
    ]
  }
)

The combination of format: :json and passing of an object to the body really kinda makes this request seem like it's going to send a JSON payload. But to the unsuspecting eye, it can inadvertently lead to a lot of Bobs getting their work calls at home.

Of course, if you RTFM, you'll understand that format: :json only applies to the request, but "Hey, RTFM" sounds like the kind of thing someone might respond to with "You must be fun at parties". And well, this is a HTTParty, right? 😉

[JangoSteve]

FYI, I recently ran into something similar, but I think one of the issues with the post as described here is that you're serializing with HTTParty::HashConversions.to_params but then parsing with CGI.parse. Those two have very different ways of thinking about serializing and de-serializing nested hashes to and from query strings, respectively, so they're not really inverses of each other.

The HTTParty::HashConversions.to_params serializes nested hashes and arrays the Rails way, and Rails uses Rack::Utils.parse_nested_query to then marshal those hashes and arrays back from the query strings.

Meanwhile, CGI.parse would be the way to marshal back hashes and arrays from a query string that was serialized instead with URI.encode_www_form.

I.e. with:

obj = {
  "name" => "Bob",
  "phones" => [
    {
      "type" => "home"
    },
    {
      "type" => "work",
      "phone" => "333-333-3333"
    }
  ]
}

This works:

obj == Rack::Utils.parse_nested_query(HTTParty::HashConversions.to_params(obj))
#=> true

And so does this:

obj == CGI.parse(URI.encode_www_form(obj))
#=> false, but close (name is now an array and strings are escaped, but home/work aren't mixed up)

But if you mix the two, you're going to have a bad time. In other words, if you use parsed_response instead of parsing it yourself with CGI.parse, or you parse it yourself with the Rack::Utils.parse_nested_query, it keeps everything aligned. If you need HTTParty to serialize things so that they can be properly parsed with CGI.parse, you can always pass the query in yourself as a string (in which case HTTParty skips the hash conversion), and serialize the string yourself with URI.encode_www_form.

All that said, I was just thinking of submitting a pull request that allows passing to HTTParty your own conversion method.

[jnunemaker]

@JangoSteve thanks for researching that. I'd love a PR.

[RoseAndres]

+1 here for a PR

Running into a similar issue trying to submit requests to a Stripe API endpoint. When sending an array of line_items that have different internal keys, it results in params that aren't compatible with their endpoints. I also don't think that changing the request format to json is an option for me, as their API accepts x-www-form-urlencoded content.

Because of this, I think that providing one's own conversion method is probably the solution we'd have to take for our use case.