Not able to update GEM_PATH Environment variable

[RamHaridas]

Describe the bug

I have installed gems using fluent-gem install --install-dir to a custom path /opt/additionalgems. I am updating the GEM_PATH environment variable with this path, but instead of loading gems installed in this directory, fluentd is still picking up gems from the default path.

To Reproduce

Install any gem to a custom directory -- fluent-gem install --install-dir /opt/additinalgems fluent-plugin-datadog -v 0.14.4

Update the GEM_PATH environment variable with customa path GEM_PATH=/opt/additionalgems:/path/to/default/gems

Run fluentd with a configuration that uses this gem, in our example @type datadog

/usr/sbin/fluentd -c /path/to/fluent.conf

This gives an error that datadog not found.

Expected behavior

Ideally it is expected that fluentd loads the gems installed in the custom directory as well, but in this case it is failing to do so. Regardless of the path addition, it still loads the gems from the default path only.

Your Environment

- Fluentd version: 1.19.0
- Package version: 6.0.0 
- Operating system: Rocky Linux 8
- Kernel version: 5

Your Configuration

<source>
      @type exec
      tag test.echo
      command echo '{"message":"Hello there, from fluentd!"}'
      run_interval 10s
      keys none
    </source>

    <match test.echo>
      @type datadog
      @id awesome_agent
      api_key YOUR_DATADOG_API_KEY
      service fluentd
      dd_source fluentd
      dd_tags env:dev
    </match>

Your Error Log

Fluent config error is thrown with message datadog not found

Additional context

Root cause of this problem is identified. It is the problem with the wrapper script at /usr/sbin/fluentd. The wrapper hard-codes the value for environment variable GEM_PATH to default location. This basically overrides the value set by user and causing fluentd to load only gems saved at default path.

Below given is the wrapper script:

#!/opt/fluent/bin/ruby
ENV["GEM_HOME"]="/opt/fluent/lib/ruby/gems/3.4.0/"
ENV["GEM_PATH"]="/opt/fluent/lib/ruby/gems/3.4.0/"   # ---> HARD-CODED
ENV["FLUENT_CONF"]="/etc/fluent/fluentd.conf"
ENV["FLUENT_CONF_INCLUDE_DIR"]="/etc/fluent/conf.d"
ENV["FLUENT_PLUGIN"]="/etc/fluent/plugin"
ENV["FLUENT_SOCKET"]="/var/run/fluent/fluentd.sock"
if ARGV.include?("--version")
  require "/opt/fluent/share/config"
  Dir.glob("/opt/fluent/lib/ruby/**/gems/**/fluent/version.rb").each do |v|
    require v.delete_suffix(".rb")
  end
  puts "fluent-package #{PACKAGE_VERSION} fluentd #{Fluent::VERSION} (#{FLUENTD_REVISION})"
  exit 0
end

if RUBY_PLATFORM =~ /linux/
  prevent_duplicate_launch = system("systemctl", "is-active", "fluentd", out: IO::NULL)
  if prevent_duplicate_launch
    if ["-c", "--config", "--dry-run", "-h", "--help", "--show-plugin-config"].none? {|allowing_opt| ARGV.include? allowing_opt}
      puts("Error: Can't start duplicate Fluentd instance with the default config.")
      if ARGV.include?("-v")
        puts("To take the version, please use '--version', not '-v' ('--verbose').")
      end
      puts <<EOS
To start Fluentd, please do one of the following:
(Caution: Please be careful not to start multiple instances with the same config.)
- Stop the Fluentd service 'fluentd'.
- Specify the config path explicitly by '-c' ('--config').
EOS
      exit 2
    end
  end
end

load "/opt/fluent/bin/fluentd"

When i run /opt/fluent/bin/fluent-gem list, I can see the datadog gem in the list, because it reads the GEM_PATH env set by the user, which is not the case with /usr/sbin/fluentd.

Is there a workaround to overcome this without having to edit the wrapper script or needing to replace it with /opt/fluent/bin/fluentd binary. Or is there a parameter that works as a replacement for GEM_PATH environment variable?

[RamHaridas]

Any reply for this would me much appreciated? 🙂

[Watson1978]

I assume this isn't intended for installing plugins in other locations. Can you let us your use case?

[RamHaridas]

The idea is to have 2 gem directories, one default and other custom. Default gem directory /opt/fluent/lib/ruby/gems/3.4.0/ is where fluent-package gems are installed. And custom gem directory is where the additional/optional gem goes, fluent-gem install --install-dir /path/to/custom

Now the objective is to let user choose if they want to load gems from /path/to/custom by setting the GEM_PATH environment variable. But the fact that wrapper hard-codes GEM_PATH to /opt/fluent/lib/ruby/gems/3.4.0/, stops us from achieving that. Unless we either modify wrapper or switch to /opt/fluent/bin/fluentd binary.

I wanted to see if there is any other workaround instead of having to modify wrapper (/usr/sbin/fluentd) or switching to /opt/fluent/bin/fluentd binary ?

[Watson1978]

The concept of allowing users to manage installed plugins in a separate directory is interesting but likely challenging to implement.

Fluent Package 5 uses Ruby 3.2. Fluent Package 6 uses Ruby 3.4. These versions lack ABI compatibility.

If you install a C-language library for Ruby 3.2, it cannot be used with Ruby 3.4. You would need to reinstall it.

Using the installation method you described, it's unclear which Ruby version the package was installed for. /opt/additionalgems/gems/zstd-ruby-1.5.7.1

You installed it for Ruby 3.2, but you might later upgrade the Fluent Package and attempt to continue using it with Ruby 3.4. This would result in errors.

I think few users would properly understand and handle this kind of scenario correctly...

[RamHaridas]

Yes, this expectation of fluentd gems to have same ruby version is well understood. Hence it is ensured that the gems installed in the custom directory will have same version as that of the gems at default path.

I just wanted to confirm that the hard coding of GEM_PATH in wrapper is intended or not. Because if that is intended, then it leaves us with no option other than to modify the wrapper script (usr/sbin/fluentd) for updating the GEM_PATH environment variable.

[Watson1978]

I just wanted to confirm that the hard coding of GEM_PATH in wrapper is intended or not. Because if that is intended, then it leaves us with no option other than to modify the wrapper script (usr/sbin/fluentd) for updating the GEM_PATH environment variable.

I confirmed with my colleague, and it's apparently intentional. This is because when installed in an arbitrary location, it becomes unmanageable by fluent-package

[RamHaridas]

Thank you for confirming this. 🙂