Unstable behavior with Ruby 3.3.1 and Process.warmup

[arekt]

trafficstars

Is this a support request? Not sure we need it anymore. We had to remove all feature flags from our background jobs as application was not working reliably. It took time to find out if the problem is related to Sidekiq, this library, Ruby or our code.

Describe the bug Related to: https://github.com/sidekiq/sidekiq/issues/6279 When gem is used in the Ruby 3.3.1 application after Process.warmup you can not get feature flags reliably.

To reproduce cat reproduce_problem.rb

# frozen_string_literal: true

require "dotenv"
require "active_support/all"           # <-------- It is important to load some bigger gem before  --->
require "launchdarkly-server-sdk"

Dotenv.load

class FFlagService
  attr_reader :client, :context
  def initialize
    ld_logger = Logger.new(STDOUT)
    ld_logger.level = Logger::DEBUG
    ld_config = LaunchDarkly::Config.new({logger: ld_logger})
    @client = LaunchDarkly::LDClient.new(ENV.fetch("LAUNCH_DARKLY_SDK_KEY"), ld_config)

    context_hash = {
      key: "anonymous",
      kind: "user",
      anonymous: true
    }

    @context = LaunchDarkly::LDContext.create(context_hash)
  end

  def enabled?(flag_key, default_value = false)
    @client.variation(flag_key, @context, default_value)
  end
end

@flags = FFlagService.new

Process.warmup

result = @flags.enabled?('FLAG_NAME_HERE')
puts result

cat Gemfile

➜ cat Gemfile
source "https://rubygems.org"

ruby "3.3.1"

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.1.3", ">= 7.1.3.2"

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"

# Use sqlite3 as the database for Active Record
gem "sqlite3", "~> 1.4"

# Use the Puma web server [https://github.com/puma/puma]
gem "puma", ">= 5.0"

# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem "importmap-rails"

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"

# Use Redis adapter to run Action Cable in production
gem "redis", ">= 4.0.1"
gem "sidekiq"
gem "dotenv"
gem "launchdarkly-server-sdk", "~> 8.4.2"
#gem "launchdarkly-server-sdk", "< 8.0.0"

# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
# gem "bcrypt", "~> 3.1.7"

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[ windows jruby ]

# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false

# Use Active Storage variants [https://guides.rubyonrails.org/active_storage_overview.html#transforming-images]
# gem "image_processing", "~> 1.2"

group :development, :test do
  # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
  gem "debug", platforms: %i[ mri windows ]
end

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem "web-console"

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"
end

group :test do
  # Use system testing [https://guides.rubyonrails.org/testing.html#system-testing]
  gem "capybara"
  gem "selenium-webdriver"
end

Expected behavior A clear and concise description of what you expected to happen.

Logs

➜ while :; do sleep 1; bundle exec ruby reproduce_problem.rb | grep -q "Unknown feature flag" && echo "error" || echo "ok"; done
ok
error
error
ok
ok
ok
ok
ok
error
ok
ok
ok
ok
error
error
error
ok
ok
ok
ok
ok
ok
ok
error
ok
error
ok
error
error
ok
ok
ok
error
ok
ok
ok
ok
error
ok
ok
error
ok
ok
ok
error
error
error
error
ok
ok
ok
ok
ok
ok
ok
ok
ok
error
ok
ok
ok
ok
ok
ok
error
error
ok
error
error
ok
error
ok
error
ok
ok
ok
ok
ok
ok
ok
error
ok
ok
ok
ok
error
ok
error
ok
ok
ok
ok
ok
ok
ok
ok
error
error
ok

SDK version gem "launchdarkly-server-sdk", "~> 8.4.2" (It happens with earlier 7.* as well.

Language version, developer tools Ruby 3.3.1

OS/platform MacOS 14.5 M1/M2 Ubuntu 2204 amd64

Additional context Process.warmup in Ruby 3.3 is a new feature and probably some extra work is required to make this gem work reliably.

[keelerm84]

Thank you for bringing this to our attention.

The SDK is not designed to be initialized pre-fork and then used post-fork. This is because the SDK creates a thread to retrieve flag data, but that thread isn't copied when the process is forked. If the process is forked before that data is retrieved, the forked copy will never have any flag data. If it is forked after the data is retrieved, it will always have stale data.

I will look into the changes surrounding Process.warmup and see what we can do to better support that change.

Again, thank you for letting us know!

[arekt]

@keelerm84 Thank you for looking at this. After your description looks like maybe it would be nice to have option to start event retrieving thread after delay or first request. Ideally Ruby or Rails in our case Sidekiq would have callback after warmup you can use for this. I see the value to be able to use Process.warmup even without threads or fork like in example above.

[dalizard]

@arekt I believe as a temporary workaround solution you could re-initialize the LaunchDarkly client in the startup Sidekiq hook, since it runs after Process.warmup e.g.:

Sidekiq.configure_server do |config|
  config.on(:startup) do
    # Re-initialize LD...
  end
end

[nerdrew]

@arekt Are you using sidekiq with forking?

[keelerm84]

I believe this issue has been resolved with the latest release (v8.8.1)

See my comment here for more information about this fix.

Thank you all for your patience and please feel free to re-open this issue if you find this does not resolve the problems you are experiencing.