ohler55
problem with serializing strong params (3.11.0+)
Problem
Hello there
Seems like I have an issue with oj (after updating to 3.16.4) not properly being used in Rails when it comes to ActionController::Parameters.
Since the users params is passed further to a Sidekiq worker, Sidekiq uses JSON.generate(object) to serialize parameters.
JSON.generate is one of three different ways of doing this incorrectly:
users = params["users"]
#=> [#<ActionController::Parameters {"email"=>"[email protected]", "first_name"=>"joe", "last_name"=>"test"} permitted: false>]
JSON.generate users
#=> "[\"{\\\"email\\\"=>\\\"[email protected]\\\", \\\"first_name\\\"=>\\\"joe\\\", \\\"last_name\\\"=>\\\"test\\\"}\"]"
JSON.dump users
#=> "[\"{\\\"email\\\"=>\\\"[email protected]\\\", \\\"first_name\\\"=>\\\"joe\\\", \\\"last_name\\\"=>\\\"test\\\"}\"]"
Oj.generate users
#=> "[\"{\\\"email\\\"=>\\\"[email protected]\\\", \\\"first_name\\\"=>\\\"joe\\\", \\\"last_name\\\"=>\\\"test\\\"}\"]"
None of those approaches returns a properly serialized object. If you try to parse it back as JSON, it will return:
reverse = Oj.load(Oj.generate users)
#=> ["{\"email\"=>\"[email protected]\", \"first_name\"=>\"joe\", \"last_name\"=>\"test\"}"]
So now, instead of an array of hashes, we end up with an array of strings. And those strings - mind you - cannot be further parsed:
reverse.map {|x| Oj.load(x)}
#=> Oj::ParseError: unexpected character (after email) at line 1, column 9 [parse.c:762]
The only module/method combination that works is:
Oj.dump users
#=> "[{\"email\":\"[email protected]\",\"first_name\":\"joe\",\"last_name\":\"test\"}]"
Oj.load _
#=> [{"email"=>"[email protected]", "first_name"=>"joe", "last_name"=>"test"}]
Question
So here's my question: Is there an option for Oj (3.16.4) and Rails 6.1+ that I'm missing?
This was working as expected up to oj 3.10.18 and it breaks with anything over that.
My current config for Oj is this:
MultiJson.use(:oj)
Oj.default_options = {
bigdecimal_as_decimal: true,
bigdecimal_load: :auto,
mode: :custom,
second_precision: ActiveSupport::JSON::Encoding.time_precision,
time_format: :xmlschema,
use_as_json: true,
}
Oj.optimize_rails
It looks like ActionController::Parameters is not encoding as a json object. I'm not really set up to test this right now so would you be able to help?
The first thing to try would be to see if there is an #as_json or #to_json method on ActionController::Parameters. If so that does it return?
prying into the controller (using 3.11.8)
params
#=> #<ActionController::Parameters {"url"=>"(...)", "users"=>[#<ActionController::Parameters {"email"=>"[email protected]", "first_name"=>"joe", "last_name"=>"test"} permitted: false>], "uuid"=>"something-something", "format"=>"json", "controller"=>"api/v1/users", "action"=>"create"} permitted: false>
users = params[:users]
#=> [#<ActionController::Parameters {"email"=>"[email protected]", "first_name"=>"joe", "last_name"=>"test"} permitted: false>]
users.as_json
#=> [{"email"=>"[email protected]", "first_name"=>"joe", "last_name"=>"test"}]
users.to_json
#=> "[{\"email\":\"[email protected]\",\"first_name\":\"joe\",\"last_name\":\"test\"}]"
and yet, after sending users to a sidekiq worker:
users
#=> ["{\"email\"=>\"[email protected]\", \"first_name\"=>\"joe\", \"last_name\"=>\"test\"}"]
Is this what you needed?
What I'm trying to determine is if the ActionController::Parameters.as_json exists and is being called. It kind of looks like #to_json is being called by Oj instead of #as_json.
What does the json gem emit when calling JSON.generate without Oj?
It appears as if the #to_json method is being called on the params instead of #as_json. It's as if the :use_as_json option is not set which would be the normal case for the JSON gem.
I'm having the same issue. I can confirm that version 3.10.18 does not have this issue. Setting Oj.default_options = { mode: :compat, use_as_json: true } in a later version also doesn't fix it nor does Oj.default_options = { mode: :compat, compat_bigdecimal: true }.
