cloudinary_gem icon indicating copy to clipboard operation
cloudinary_gem copied to clipboard

BlobKey, MirrorService and Sidekiq 8 are incompatible

Open adampope opened this issue 7 months ago • 3 comments

Bug report for Cloudinary Ruby SDK

Describe the bug in a sentence or two.

When using the Cloudinary ActiveStorage service in a mirror with S3 and Sidekiq 8 as a job queue, I am getting errors of the form

Job arguments to ActiveStorage::MirrorJob must be native JSON types, but "1xfiunhqeexf8h3pkdxucnby5eou" is a ActiveStorage::BlobKey

Issue Type (Can be multiple)

  • [ ] Build - Cannot install or import the SDK
  • [ ] Performance - Performance issues
  • [X] Behaviour - Functions are not working as expected (such as generate URL)
  • [ ] Documentation - Inconsistency between the docs and behaviour
  • [ ] Other (Specify)

Steps to reproduce

  • Use Sidekiq 8 with strict_args enabled (default)
  • Define an S3 service and a Cloudinary Service in storage.yml
  • Define a mirror as
mirror:
  service: Mirror
  primary: amazon
  mirrors:
    - cloudinary
  • Attempt to upload a file
  • ActiveStorage will attempt to enqueue ActiveStorage::MirrorJob which takes key as an arg. But because the Cloudinary gem is redefining the key as a BlobKey we get the above error from Sidekiq because BlobKey is not a native JSON type.

Error screenshots or Stack Trace (if applicable)

lib/active_storage/service/mirror_service.rb:55

A minimal repro can be achieved with the following:

> s = ActiveStorage::Service::MirrorService.new(primary: nil, mirrors: [])
=> #<ActiveStorage::Service::MirrorService:0x0000ffff69a5d2f0 @mirrors=[], @primary=nil>
> k = ActiveStorage::BlobKey.new(key: "test")
=> "test"
> s.mirror_later(k, checksum: nil)
INFO  2025-05-15T14:47:39.511Z pid=955 tid=25v: Sidekiq 8.0.3 connecting to Redis with options {size: 10, pool_name: "internal", url: "redis://redis:6379/0"}
Failed enqueuing ActiveStorage::MirrorJob to Sidekiq(default): ArgumentError (Job arguments to ActiveStorage::MirrorJob must be native JSON types, but "test" is a ActiveStorage::BlobKey.
See https://github.com/sidekiq/sidekiq/wiki/Best-Practices
To disable this error, add `Sidekiq.strict_args!(false)` to your initializer.

This monkey patch to ActiveStorage can address the issue, but I dont know whether rails would want this in their core code when the expect the key to be a string anyway

module ActiveStorage
  class Service::MirrorService < Service
    def mirror_later(key, checksum:) # :nodoc:
      ActiveStorage::MirrorJob.perform_later key.to_s, checksum: checksum
    end
  end
end

Operating System

  • [X] Linux
  • [ ] Windows
  • [ ] macOS
  • [ ] All

Environment and Libraries (fill in the version numbers)

  • Cloudinary Ruby SDK version - 2.3.0
  • Ruby Version - 3.4.3
  • Rails Version - 7.2.2.1
  • Other Libraries - ActiveStorage 7.2.2.1

adampope avatar May 15 '25 14:05 adampope

@adampope we need to research thsi but did you try this: To disable this error, add Sidekiq.strict_args!(false) to your initializer.

tommyg-cld avatar May 27 '25 09:05 tommyg-cld

@tommyg-cld I did not. That would definitely work as it restores the old behaviour but goes against the default and recommended way to use Sidekiq.

From the sidekiq 7.0 release notes

Strict argument checking is now enabled by default. Sidekiq will raise an error if you pass a Symbol or generic Ruby object into perform_async because they do not serialize correctly into JSON. This has long been a source of frustration and debugging time for users, better to be upfront about this design decision. From the beginning, Sidekiq has used plain JSON as its job format, for interoperability with all programming languages and environments.

My monkey patch has limited the scope of the fix to Cloudinary/Active Storage, whereas disabling strict args could allow more errors in across the code base.

adampope avatar May 27 '25 09:05 adampope

@adampope we'll investigate further and update you

tommyg-cld avatar May 29 '25 16:05 tommyg-cld