Recent versions (3.11.4 and 3.11.5) are causing issues with the Pact library

[iamvery]

👋 I'll say up front that this was difficult to narrow down, and I'm still not certain of the root cause. It's unclear to me whether it's ultimately on the Oj or Pact side.

Problem

After updating to Oj 3.11.4 and 3.11.5, we are seeing intermittent test failures for Pact examples. The failures appear to be a result of how objects are serialized. On oj 3.11.3, responses present like:

{ "some_field" => #<Pact::SomethingLike...> }

But after upgrading certain responses later in test runs present like:

{ "some_field" => { "json_class" => "Pact::SomethingLike", ... }

Note: the issue is currently presenting itself for certain RSpec "seeds" and often happens after some successfully and correctly serialized response bodies. For example, the first Pact test will run correctly and successfully and a subsequent one is serialized incorrectly as shown above which results in failure.

Thank you for your work on this library, and please let me know if I can provide more information. Unfortunately at this time, I'm not certain how to reproduce the issue outside of our code and test suite, but I'm hopefully the reporting it may trigger some thought on recent changes.

» ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin19]
» rails -v
Rails 6.1.3.1

See also https://github.com/pact-foundation/pact-ruby/issues/240

[ohler55]

The changes from 3.11.4 to 3.11.5 were all centered around the Oj.generate() call. In the older version that could would not make sure that mimic_JSON was called. The latest version does. Are you using that call in you code or do you know if the Pact library is?

[iamvery]

The changes from 3.11.4 to 3.11.5 were all centered around the Oj.generate() call. In the older version that could would not make sure that mimic_JSON was called. The latest version does. Are you using that call in you code or do you know if the Pact library is?

We installed Oj in order to speed up our use of the Blueprinter library (docs). I believe they use Oj.generate. That adds another interesting layer to this problem. I'm not sure what Pact is doing, but I recall references to generate in that library as well. Does Oj perform any monkey patches of other libraries/code?

[ohler55]

Oj does monkey patch when in mimic mode. It has to to act like the JSON gem. That is the only case though. Normally I avoid money patching like a plague.

[iamvery]

Oj does monkey patch when in mimic mode. It has to to act like the JSON gem. That is the only case though. Normally I avoid money patching like a plague.

We may be onto something with that. It wasn't clear to me from the docs, how are modes set? Does it happen automatically when used with Rails (for example)? Does the mode change implicitly for any reason? I'm trying to think through how/why the behavior that I'm seeing is intermittent on different test orderings.

[ohler55]

The default options should not change but rails mode does call Oj.mimic_JSON and that does change the JSON gem monkey patches.

One thought on the different order might be the inclusion or rails or the JSON gem. The order of either could result in different patches being at the forefront.

[ohler55]

I should have answered your question about setting mode. That can be set an an optional argument or in the default_opttions.

[zorab47]

@ohler55 is there an Oj method to show its current configuration? If I disable the use of Oj in our app there are no issues. Calling Oj.optimize_rails() specifically raises the issue again. I'd like to see if something is changing between the test runs (like order of code loading).

I'm working with @iamvery to resolve the issue.

[zorab47]

I've reproduced the issue in the Ruby file oj-blueprinter-pact-reproduction-script.rb. The issue is likely due to an interaction the Blueprinter library is doing with Oj. It will require additional research.

The top of the file is set up to toggle between Oj versions 3.11.3 and 3.11.5 to show the issue when Oj 3.11.5 is used.

https://gist.github.com/zorab47/f00e4ad8d5b02b583c2ba97db6011f67

cc @bethesque

[zorab47]

Quick update: After checking the Blueprinter library I was able to reproduce without that dependency. The gist is now updated to reproduce the issue without Blueprinter.

The key order to reproduce the issue is to

1. Call Oj.generate early
2. Execute the pacts to reproduce the failure on version 3.11.5

[ohler55]

You can check the default options Oj.default_options but that does not tell you if Oj.mimic_JSON or Oj.optimize_rails has been called.

As for the calling of generate, generate is meant to mimic the JSON gem so it calls mimic_JSON which, like the JSON gem monkey patches core objects. A horrible approach but none the less it is what it required to be compatible. I would suggest that unless you want to live with the monkey patched version, don't call generate but call the alternative Oj methods such as load, to_stream, or to_file.

[bethesque]

@ohler55 do you have any suggestions for how Pact can safeguard against other gems that modify the JSON behaviour, so that it always calls the default JSON? This is where the JSON gets created. https://github.com/pact-foundation/pact-mock_service/blob/b06f88f75a92df4bd97a8c0d6a40ea5d62adf341/lib/pact/consumer_contract/consumer_contract_writer.rb#L75

There are already a bunch of hacks in there to help it deal with the Stomping Feet of Active Bloody Support.

[ohler55]

What generally causes the problems is counting on the objects themselves to encode themselves correctly. Rails and the JSON don't agree yet both monkey patch the core objects. A mess as you have seen. The only way around that is to not allow the objects to encode themselves. Oj let you do that by turning off to_json and as_json calls to the objects. With those turned off the behaviour is totally up to you. For Oj avoid generate and stick with one of the other modes. The custom mode give you the most flexibility.

[bethesque]

Actually, now I've had another look at the failing example, I think it's the JSON load that is causing the issues, not the JSON generate.

[ohler55]

Let me know what you find or what else I can do to help.

[zorab47]

Is there a way to detect when the mimic JSON compatibility mode is being used? Is there a configuration to disable it?

[ohler55]

There is not an easy way. There might be some way of checking symbols and classes but nothing obvious. The way to disable is to not call the to_json or as_json methods on objects. That can be done by calling Oj.dump with the mode set to something like :custom and making sure :use_to_json and :use_as_json are false.

[ohler55]

Okay to close?

[iamvery]

This is still a problem for us. We have been locked to a specific version since the issue came up. I tried updating to the latest today and still see the same error. It looks like there are similar/related issues in #659

[iamvery]

I tried removing our lock on Oj and running things again, and it appears that something in a recent release has fixed the issue.

[iamvery]

I spoke too soon. The failures are order dependent so a passing build gave me false confidence 😞

[ohler55]

What is the status of this issue? Are you seeing differences between the JSON gem and Oj intermittently?

[iamvery]

I am not aware of us revisiting this, but it was still an issue the last time we looked at it.

[ohler55]

I'll take another look then.

[zorab47]

Revisiting this issue again. 👋

I am able to resolve the issue by setting Oj.default_options = { create_additions: true } after Oj enters :compat mode. Was there a behavior change in how "create additions" functioned after v3.11.3?

I've updated the Gist with an example: https://gist.github.com/zorab47/f00e4ad8d5b02b583c2ba97db6011f67#file-oj-pact-reproduction-script-rb-L106-L126