apm-agent-ruby icon indicating copy to clipboard operation
apm-agent-ruby copied to clipboard
verified publisher icon elastic

Errno::EROFS thrown when run from a docker container with a RO filesystem

Open smedrick opened this issue 1 year ago • 0 comments

Describe the bug

When running our Rails app on a docker container with a read-only Filesystem mount, we get

`/home/overlay` is not writable.
/usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:117:in `rescue in filesystem_access': There was an error accessing `/app`. (Bundler::GenericSystemCallError)
The underlying system error is Errno::EROFS: Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:102:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'
/usr/share/ruby/tmpdir.rb:89:in `mkdir': Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135 (Errno::EROFS)
	from /usr/share/ruby/tmpdir.rb:89:in `block in mktmpdir'
	from /usr/share/ruby/tmpdir.rb:135:in `create'
	from /usr/share/ruby/tmpdir.rb:87:in `mktmpdir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:315:in `tmp'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:682:in `block in tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:103:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'

I suspect this is the commit that breaks compatibility with a RO FS: https://github.com/elastic/apm-agent-ruby/commit/23b4e4d8961459877025678d17c9cac0a68c4da9

and browsing Issue history on the Bundler repo, it appears default_bundle_dir was not designed with the intent that it would be used outside of the install process.

Steps to reproduce

  1. Create Rails app with latest version of apm agent
  2. Build docker image with pre-bundled Rails app
  3. Deploy docker image to Kubernetes cluster using helm directive
        securityContext:
          readOnlyRootFilesystem: true

Expected behavior

Able to run bundle exec rails c or bundle exec rails s without encountering exception

Environment

  • OS: Red Hat Universal Base Image 8
  • Ruby version: 2.6.4
  • Framework and version: Rails 5.2.8, Bundler 2.2.33
  • APM Server version:
  • Agent version: 4.7.3

Additional context

Add any other context about the problem here.

  • Agent config options
    Click to expand 
      enabled: false
  log_path: log/elastic-apm-<%= ENV['HOSTNAME'] %>.log
  service_node_name: <%= ENV['HOSTNAME'] %>

smedrick avatar Jul 02 '24 20:07 smedrick