dry-system icon indicating copy to clipboard operation
dry-system copied to clipboard

provider problem in multi threaded condition

Open FLemon opened this issue 8 months ago • 7 comments

Describe the bug

when using dry-system in multi threaded platform, i.e. puma. when lazy loading provider there's race condition in the provider start block

in our production environment, we are getting key not found error when calling Container['path.to.key]`

[129] - Worker 0 (PID: 139) booted in 2.14s, phase: 0
Dry::Core::Container::KeyError: key not found: "decorators.decorated_entities.users.with_pending_changes"
Did you mean?  "decorators.builders.users.with_pending_changes"
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/resolver.rb:32:in `block in call'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/resolver.rb:28:in `fetch'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/resolver.rb:28:in `call'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/mixin.rb:132:in `resolve'
        /usr/local/bundle/gems/dry-system-1.0.1/lib/dry/system/container.rb:493:in `resolve'
        /usr/src/app/system/container.rb:151:in `resolve'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/mixin.rb:145:in `[]'
        /usr/src/app/system/container.rb:137:in `[]'
        /usr/local/bundle/gems/dry-auto_inject-1.0.1/lib/dry/auto_inject/strategies/kwargs.rb:16:in `block (3 levels) in define_new'
        /usr/local/bundle/gems/dry-auto_inject-1.0.1/lib/dry/auto_inject/strategies/kwargs.rb:15:in `each'
        /usr/local/bundle/gems/dry-auto_inject-1.0.1/lib/dry/auto_inject/strategies/kwargs.rb:15:in `block (2 levels) in define_new'
        /usr/local/bundle/gems/dry-system-1.0.1/lib/dry/system/loader.rb:54:in `call'
        /usr/local/bundle/gems/dry-system-1.0.1/lib/dry/system/component.rb:64:in `instance'
        /usr/local/bundle/gems/dry-system-1.0.1/lib/dry/system/container.rb:639:in `block in load_local_component'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/item/memoizable.rb:35:in `block in call'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/item/memoizable.rb:34:in `synchronize'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/item/memoizable.rb:34:in `call'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/resolver.rb:36:in `call'
        /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/mixin.rb:132:in `resolve'
        /usr/local/bundle/gems/dry-system-1.0.1/lib/dry/system/container.rb:493:in `resolve'

we tried to replicate the exact issue we are having by putting a new spec in this repo(as shown below), but all we can see is warning of circular require(as shown in the stacktrace below)

the warning kinda illustrated the problem.

/usr/local/bundle/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:34: warning: /usr/local/bundle/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:34: warning: loading in progre
ss, circular require considered harmful - /tmp/d20240603-1904-aagks2/lib/animals/cat.rb
        from /usr/src/app/spec/integration/container/providers/resolving_root_key_spec.rb:56:in  `block (5 levels) in <top (required)>'
        from /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/mixin.rb:145:in  `[]'
        from /usr/src/app/lib/dry/system/container.rb:493:in  `resolve'
        from /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/mixin.rb:132:in  `resolve'
        from /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/resolver.rb:36:in  `call'
        from /usr/local/bundle/gems/dry-core-1.0.1/lib/dry/core/container/item/callable.rb:16:in  `call'
        from /usr/src/app/lib/dry/system/container.rb:639:in  `block in load_local_component'
        from /usr/src/app/lib/dry/system/component.rb:64:in  `instance'
        from /usr/src/app/lib/dry/system/loader.rb:47:in  `call'
        from /usr/src/app/lib/dry/system/loader.rb:33:in  `require!'
        from /usr/local/bundle/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:34:in  `require'
        from /usr/local/bundle/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:34:in  `require'

To Reproduce

spec put in spec/integration/providers/resolving_root_key_spec.rb

  context "lazy loading" do
    it "is thread safe" do
      threads = []
      2.times do
        threads << Thread.new do
          Test::Container['animals.cat']
        end
      end 
      expect { threads.each(&:join) }.not_to raise_error
    end

Expected behavior

Expected running(or triggering) of the provider steps are thread safe

My environment

  • Affects my production application: YES
  • Ruby version: 3.0.6
  • OS: Linux/MacOS

FLemon avatar Jun 03 '24 15:06 FLemon