fluentd
fluentd copied to clipboard
Published
20 hours ago •
fluent
fluent
Loading same plugin from multiple directories cause unfriendly error
Describe the bug
Fluentd raises Fluent::Plugin::RegexpParser: description specified twice for expression (ArgumentError) when using a "parser" filter. Tested on 1.12.2 and 1.7.0
To Reproduce
I used the same configuration as given in the example: https://docs.fluentd.org/filter/parser
<source>
@type tail
path /any/file.log
tag foo.bar
</source>
<filter foo.bar>
@type parser
key_name log
<parse>
@type regexp
expression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
</filter>
Expected behavior It should run, or at least give an understandable error if I made a mistake
Your Environment
- Fluentd or td-agent version:
fluentd 1.12.2andfluentd 1.7.0 - Operating system:
NixOS - Kernel version:
5.7.11
If you hit the problem with older fluentd version, try latest version first.
Your Error Log
`2021-03-31 12:53:08 +0200 [info]: parsing config file is succeeded path="fluentd2.conf"
Traceback (most recent call last):
30: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/bin/fluentd:20:in `<main>'
29: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/bin/fluentd:20:in `load'
28: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/bin/fluentd:8:in `<top (required)>'
27: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/bin/fluentd:8:in `require'
26: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/command/fluentd.rb:314:in `<top (required)>'
25: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/supervisor.rb:504:in `run_supervisor'
24: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/supervisor.rb:599:in `supervise'
23: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/supervisor.rb:581:in `dry_run'
22: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/supervisor.rb:804:in `run_configure'
21: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/engine.rb:96:in `run_configure'
20: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/engine.rb:131:in `configure'
19: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/root_agent.rb:156:in `configure'
18: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/root_agent.rb:156:in `each'
17: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/root_agent.rb:160:in `block in configure'
16: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/root_agent.rb:315:in `add_source'
15: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin.rb:100:in `new_input'
14: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin.rb:146:in `new_impl'
13: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/registry.rb:44:in `lookup'
12: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/registry.rb:86:in `search'
11: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/registry.rb:86:in `require'
10: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:23:in `<top (required)>'
9: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:23:in `require'
8: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_multiline.rb:18:in `<top (required)>'
7: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_multiline.rb:18:in `require'
6: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_regexp.rb:19:in `<top (required)>'
5: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_regexp.rb:20:in `<module:Fluent>'
4: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_regexp.rb:21:in `<module:Plugin>'
3: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_regexp.rb:25:in `<class:RegexpParser>'
2: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/configurable.rb:158:in `config_param'
1: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/config/configure_proxy.rb:321:in `config_param'
/nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/config/configure_proxy.rb:345:in `config_set_desc': Fluent::Plugin::RegexpParser: description specified twice for expression (ArgumentError)
The log for 1.12.2:
2021-03-31 13:14:16 +0200 [info]: parsing config file is succeeded path="fluentd2.conf"
2021-03-31 13:14:16 +0200 [info]: gem 'fluentd' version '1.12.2'
Traceback (most recent call last):
27: from ./result/bin/fluentd:20:in `<main>'
26: from ./result/bin/fluentd:20:in `load'
25: from /nix/store/w733dsdyxfsx10wfpzv8r3w7263wishh-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/bin/fluentd:8:in `<top (required)>'
24: from /nix/store/w733dsdyxfsx10wfpzv8r3w7263wishh-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/bin/fluentd:8:in `require'
23: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/command/fluentd.rb:345:in `<top (required)>'
22: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:648:in `run_supervisor'
21: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:80:in `run_configure'
20: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:105:in `configure'
19: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:152:in `configure'
18: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:152:in `each'
17: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:158:in `block in configure'
16: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:312:in `add_source'
15: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin.rb:105:in `new_input'
14: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin.rb:160:in `new_impl'
13: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/registry.rb:44:in `lookup'
12: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/registry.rb:86:in `search'
11: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/registry.rb:86:in `require'
10: from /nix/store/w733dsdyxfsx10wfpzv8r3w7263wishh-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:23:in `<top (required)>'
9: from /nix/store/w733dsdyxfsx10wfpzv8r3w7263wishh-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:23:in `require'
8: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_multiline.rb:18:in `<top (required)>'
7: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_multiline.rb:18:in `require'
6: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_regexp.rb:19:in `<top (required)>'
5: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_regexp.rb:20:in `<module:Fluent>'
4: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_regexp.rb:21:in `<module:Plugin>'
3: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/plugin/parser_regexp.rb:25:in `<class:RegexpParser>'
2: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/configurable.rb:158:in `config_param'
1: from /nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/config/configure_proxy.rb:321:in `config_param'
/nix/store/1xf5p5cb35jdxmj5zzdinx256da521rm-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/lib/fluent/config/configure_proxy.rb:345:in `config_set_desc': Fluent::Plugin::RegexpParser: description specified twice for expression (ArgumentError)
<source> @type tail path /any/file.log tag foo.bar </source> <filter foo.bar> @type parser key_name log <parse> @type regexp expression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/ time_format %d/%b/%Y:%H:%M:%S %z </parse> </filter>
Cannot reproduce with the given config.
$ bundle exec fluentd -c issue-3310.conf
2021-03-31 21:56:20 +0900 [info]: parsing config file is succeeded path="issue-3310.conf"
2021-03-31 21:56:20 +0900 [info]: gem 'fluentd' version '1.12.2'
2021-03-31 21:56:20 +0900 [error]: config error file="issue-3310.conf" error_class=Fluent::ConfigError error="<parse> section is required."
Probably you are omitting something to paste.
10: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:23:in `<top (required)>'
9: from /nix/store/na19fwwimzbiybraq8ml51w5da73404y-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:23:in `require'
8: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_multiline.rb:18:in `<top (required)>'
7: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_multiline.rb:18:in `require'
6: from /nix/store/5xxrb75cby1h2kpb1kbpm7lsps2lazry-ruby2.6.6-fluentd-1.7.0/lib/ruby/gems/2.6.0/gems/fluentd-1.7.0/lib/fluent/plugin/parser_regexp.rb:19:in `<top (required)>'
Please give us a config as possible same that you actually use.
@ashie : thanks for your response. I actually gave you the exact same configuration file as the one that produces the error. I looked at the packaging script too from nixos and saw nothing wrong (a Gemfile with only fluentd, so the result should be the same as your "bundle exec" command)
You gave a snippet of my backtrace but I don’t see anything wrong here either. Are you suspecting that something is wrong because there are two different paths? If so it might be a distribution issue I can dig into it
Some news: while trying to debug the issue I "fixed" it inadvertently. Here is what I had:
when enabling verbose mode, I obtained that log:
2021-04-01 01:34:43 +0200 [trace]: fluent/log.rb:286:trace: registered filter plugin 'parser'
2021-04-01 01:34:43 +0200 [trace]: fluent/log.rb:286:trace: registered parser plugin 'regexp'
2021-04-01 01:34:43 +0200 [trace]: fluent/log.rb:286:trace: registered parser plugin 'regexp'
Which seemed to indicate that the parser plugin 'regexp' was registered twice. The ruby environment in nixOS is built by default using symlinks, and when I replaced those symlinks with deep copy (to make it easier for me to debug), things magically started to work. I suspect that due to those symlinks, the same parser plugin "regexp" was called and registered twice because somehow ruby found it at two locations that were actually the same path (due to symlink). I didn’t have time to get familiar with the plugin registration process so please bear with me if my analysis is not exact.
That’s now easy to fix in my distribution, but maybe it would be a good thing to deduplicate the paths based on their real location when doing plugin registration? I leave that open in case you want to consider that.
Which seemed to indicate that the parser plugin 'regexp' was registered twice.
I got it. Now I can reproduce by using -p option of fluentd.
I'm not familiar with NixOS, so could you describe the concrete symlinks built by it?
The hashes and symlinks can be quite intricated, here is what I got (I shorted the paths for readability): The "fluentd" binary is a wrapper that contains that:
ENV["BUNDLE_GEMFILE"] = "/A/Gemfile"
ENV.delete 'BUNDLE_PATH'
ENV['BUNDLE_FROZEN'] = '1'
Gem.paths = { 'GEM_HOME' => "/B/lib/ruby/gems/2.6.0" }
$LOAD_PATH.unshift "/C/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib"
require 'bundler'
Bundler.setup("default")
load Gem.bin_path("fluentd", "fluentd")
The /A/ path contains the Gemfile and Gemfile.lock (only gem "fluentd") The /C/ path contains only bundler bin and library
The /B/lib/ruby/gems/2.6.0/gems is a directory which contains a symlink to each dependency (bundler-2.1.4 concurrent-ruby-1.1.8 cool.io-1.7.1 fluentd-1.12.2 http_parser.rb-0.6.0 msgpack-1.4.2 serverengine-2.2.3 sigdump-0.2.4 strptime-0.2.5 tzinfo-2.0.4 tzinfo-data-1.2021.1 webrick-1.7.0 yajl-ruby-1.4.1)
Each of those symlink targets a distinct path with the raw bundle without anything more
Similarly the /B/lib/ruby/gems/2.6.0/bin/* are symlinks to the same raw bundles
When I turn into "copy mode" (the mode that fixes the issue), those symlinks (both bin and gems) are replaced by their content, the rest doesn’t change
An "ls" might help visualize:
immae@dilion ~ $ ls -l /B/lib/ruby/gems/2.6.0/gems
total 52
lrwxrwxrwx 1 root root 96 Jan 1 1970 bundler-2.1.4 -> /D-bundler-2.1.4/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/
lrwxrwxrwx 1 root root 122 Jan 1 1970 concurrent-ruby-1.1.8 -> /E-ruby2.6.6-concurrent-ruby-1.1.8/lib/ruby/gems/2.6.0/gems/concurrent-ruby-1.1.8/
lrwxrwxrwx 1 root root 106 Jan 1 1970 cool.io-1.7.1 -> /F-ruby2.6.6-cool.io-1.7.1/lib/ruby/gems/2.6.0/gems/cool.io-1.7.1/
lrwxrwxrwx 1 root root 108 Jan 1 1970 fluentd-1.12.2 -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/gems/fluentd-1.12.2/
lrwxrwxrwx 1 root root 120 Jan 1 1970 http_parser.rb-0.6.0 -> /H-ruby2.6.6-http_parser.rb-0.6.0/lib/ruby/gems/2.6.0/gems/http_parser.rb-0.6.0/
lrwxrwxrwx 1 root root 106 Jan 1 1970 msgpack-1.4.2 -> /I-ruby2.6.6-msgpack-1.4.2/lib/ruby/gems/2.6.0/gems/msgpack-1.4.2/
lrwxrwxrwx 1 root root 116 Jan 1 1970 serverengine-2.2.3 -> /J-ruby2.6.6-serverengine-2.2.3/lib/ruby/gems/2.6.0/gems/serverengine-2.2.3/
lrwxrwxrwx 1 root root 106 Jan 1 1970 sigdump-0.2.4 -> /K-ruby2.6.6-sigdump-0.2.4/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/
lrwxrwxrwx 1 root root 108 Jan 1 1970 strptime-0.2.5 -> /L-ruby2.6.6-strptime-0.2.5/lib/ruby/gems/2.6.0/gems/strptime-0.2.5/
lrwxrwxrwx 1 root root 104 Jan 1 1970 tzinfo-2.0.4 -> /M-ruby2.6.6-tzinfo-2.0.4/lib/ruby/gems/2.6.0/gems/tzinfo-2.0.4/
lrwxrwxrwx 1 root root 120 Jan 1 1970 tzinfo-data-1.2021.1 -> /N-ruby2.6.6-tzinfo-data-1.2021.1/lib/ruby/gems/2.6.0/gems/tzinfo-data-1.2021.1/
lrwxrwxrwx 1 root root 106 Jan 1 1970 webrick-1.7.0 -> /O-ruby2.6.6-webrick-1.7.0/lib/ruby/gems/2.6.0/gems/webrick-1.7.0/
lrwxrwxrwx 1 root root 110 Jan 1 1970 yajl-ruby-1.4.1 -> /P-ruby2.6.6-yajl-ruby-1.4.1/lib/ruby/gems/2.6.0/gems/yajl-ruby-1.4.1/
immae@dilion ~ $ ls -l /B/lib/ruby/gems/2.6.0/bin
total 48
lrwxrwxrwx 1 root root 88 Jan 1 1970 bundle -> /D-bundler-2.1.4/lib/ruby/gems/2.6.0/bin/bundle*
lrwxrwxrwx 1 root root 89 Jan 1 1970 bundler -> /D-bundler-2.1.4/lib/ruby/gems/2.6.0/bin/bundler*
lrwxrwxrwx 1 root root 113 Jan 1 1970 fluent-binlog-reader -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-binlog-reader*
lrwxrwxrwx 1 root root 111 Jan 1 1970 fluent-ca-generate -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-ca-generate*
lrwxrwxrwx 1 root root 107 Jan 1 1970 fluent-cap-ctl -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-cap-ctl*
lrwxrwxrwx 1 root root 103 Jan 1 1970 fluent-cat -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-cat*
lrwxrwxrwx 1 root root 103 Jan 1 1970 fluent-ctl -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-ctl*
lrwxrwxrwx 1 root root 100 Jan 1 1970 fluentd -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluentd*
lrwxrwxrwx 1 root root 105 Jan 1 1970 fluent-debug -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-debug*
lrwxrwxrwx 1 root root 103 Jan 1 1970 fluent-gem -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-gem*
lrwxrwxrwx 1 root root 120 Jan 1 1970 fluent-plugin-config-format -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-plugin-config-format*
lrwxrwxrwx 1 root root 115 Jan 1 1970 fluent-plugin-generate -> /G-ruby2.6.6-fluentd-1.12.2/lib/ruby/gems/2.6.0/bin/fluent-plugin-generate*
I think I correctly reflected the tree, please ask if you see something that could be incorrect. Hope it help.