Importing puslar client libraries creates zombie dbus processes

[desolatorxxl]

As per discord chat with @Jeffail and @mihaitodor:

Running any of benthos subcommands in a cronjob will create /usr/bin/dbus-daemon --session processes for every invocation, slowing down the system or ultimately crashing it.

This is most likely because there are no dbus sessions available to cron processes.

Way to reproduce:

# cat /etc/cron.d/foo
* * * * * root benthos --version
# Wait a few minutes
# ps aux | grep dbus
message+     605  0.0  0.0   8572  4460 ?        Ss   11:28   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-o
root      302826  0.0  0.0   8268  2480 ?        Ss   19:16   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      303318  0.0  0.0   8268  2480 ?        Ss   19:17   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      303870  0.0  0.0   8268  2504 ?        Ss   19:18   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
... a few more ...
root      309243  0.0  0.0   8268  2480 ?        Ss   19:28   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      309737  0.0  0.0   8268  2540 ?        Ss   19:29   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      310353  0.0  0.0   8268  2484 ?        Ss   19:30   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      310842  0.0  0.0   8268  2484 ?        Ss   19:31   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

This also happens on every invocation in a regular shell on a system where dbus is disabled / not existent (e.g. OpenRC systems).

The root cause is most likely this: https://github.com/99designs/keyring/blob/81fed19d324843057dab56ba67dd0db54d1d4626/secretservice.go#L19

Workaround is to set DBUS_SESSION_BUS_ADDRESS=/dev/null as environment variable in the cron file or from where ever else you might run it.

[Jeffail]

So I've removed pulsar from the default benthos builds: https://github.com/benthosdev/benthos/commit/1792ca473d8cb731cf3b9d16da27a205a62fb1f8, and the first release of V4 (unless this gets fixed) will come without it. However, the option to compile Benthos with the standard pulsar input and output is still there, I've added an explanation and example on how to do that in the migration guide.

Ideally we need to let the pulsar client lib maintainers know. To me it seems odd that a dependency like this is even being drawn into a client library by default, but I'm not sure whether the solution would be to chase upstream and then upgrade or to bring out their authentication plugins into separate packages.

Either way I'm not comfortable importing these packages by default, we've had multiple issues with the dependencies being brought in by the pulsar client library over the last year or so and my gut feeling now is to hold back on using it.

[Jeffail]

Related: https://github.com/99designs/keyring/issues/103

[teabot]

Did this get raised with the Pulsar client lib project?

[tspannhw]

Any updates?

[Jeffail]

Hey @tspannhw, unfortunately I'm not convinced this will be addressed by the client authors any time soon so I don't think there's any convenient way in which we can progress this ourselves.

[mihaitodor]

For now, if you wish to use the Pulsar components, you should import Benthos as a library as explained here and build it using Go 1.18+. The steps would be the following:

• Download & install Go 1.18 (or use the official Docker container)
• Create a folder somewhere and place in it main.go with the contents that I linked above
• Run in that folder go build -o benthos .
• You should now have a fully-featured benthos binary including Pulsar support

[srenatus]

Could it have made a reappearance? Judging from this,

% go mod why github.com/99designs/keyring
# github.com/99designs/keyring
github.com/something/nondisclosed
github.com/benthosdev/benthos/v4/public/components/sql
github.com/snowflakedb/gosnowflake
github.com/99designs/keyring

So it's come back through gosnowflake, I suppose? 😱

[Jeffail]

hecckkkkkk, 99designs more like 99attemptstoruinyourproduct

[Jeffail]

I guess we need to give up on the idea that libraries are going to migrate away from this package, pulsar still has it as far as I can tell. I believe there's a way to manually nuke it from our deps with a replacement but needs investigation.

Either that or we add a lint check specifically for this package to try and avoid future imports.

[srenatus]

Replacements will do fine when it comes to building benthos. They will, however, not be honored when building packages that use benthos as golang library, but they'll need to be reproduced in the go.mod files there. (Here's an example, https://github.com/open-policy-agent/opa/commit/0fa850bffa9d2d915fc6cd7e2382c116a790688a, FWIW)

[Jeffail]

Okay I'm bored of waiting so I've made a fork that nukes all dbus functionality from keyring and replaced usage of it in our mod: https://github.com/benthosdev/benthos/commit/7453f87010cfc96d60cf0d484af33f1e0ac59abd

Next steps are:

• Verify that actually fixes the problem
• Consider re-adding pulsar back into the official build
• Add documentation for custom binary builders explaining that this is an issue if they choose to import pulsar or snowflake stuff and recommending they do the same replacement directive

[Jeffail]

I've added pulsar back into the main distribution of benthos: https://github.com/benthosdev/benthos/commit/01aca871171af895d7f1cb1646d18d4923acc70b, hopefully we're still good with the dependency swap. I'm still not going to be happy until we can get a non-hacky fix from upstream so leaving this open until that happens. Which it will. Surely it will. Yes, it will. Nah probably not.