[bitnami/fluentd] Add typhoeus gem

[kvokka]

Name and Version

bitnami/bluentd:latest

What is the problem this feature will solve?

This is the optional dependency of https://github.com/uken/fluent-plugin-elasticsearch#http_backend But it helps to fix this problem https://github.com/uken/fluent-plugin-elasticsearch/issues/453#issuecomment-1046401927

Also, want to mention, that the optional gem oj is included in the package, so why not install typhoeus gem? (it's pretty small)

What is the feature you are proposing to solve the problem?

install typhoeus gem

What alternatives have you considered?

No response

[fmulero]

Hi @kvokka

Thanks a lot for your suggestion. I'll open an internal discussion to evaluate your proposal. Unfortunately we have too much in our plate at the moment so we can not give you any ETA. In the meantime you could extend our image or make your own fork. PRs are welcome!

Regards

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[kvokka]

@fmulero I would be happy to make the PR, the only thing is that it does not feel right to add yet another line with

RUN fluent-gem typhoeus

in Dockerfile. In the other hand it feels that all the magic should happen in

RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack "fluentd" "1.14.5-5" --checksum 5398ceaab6ab2ee8b276fd305da5475efa96ef653316177e0e38750ab6448131

line, where all the plugins are bundled, but i can not find any reference to this component. May I kindly ask you to point out there this component is been baking?

Thank you!

[fmulero]

Hi @kvokka,

I am having a look to this container and you are completely right, all the magic happens inside of the fluentd package, it has all the plugins installed. That makes very difficult to include new plugins.

I'll share it internally, maybe is worth to review this container and implement a solution like we have in airflow

[kvokka]

Hi @fmulero

Thank you for such a quick response. There are options for how to make a workaround and make this work (and it's not a problem for me). Just mentioning, that it is kind of inconsistent to include optional gem oj in the package but ignore typhoeus. This change will not make a noticeable change in disk consumption but can add just some convenience for the end-user.

If there is an option to participate or PR to fluentd package - I'll be happy to (just do not know where the repo is hiding)

[fmulero]

Hi @kvokka

Unfortunately our compilation recipes are not public. I agree with you that having oj without typhoeus is not very useful but we are trying to avoid adding more and more plugins or addons to our images and let the users do it by themselves.

In this particular case I think you could do it downloading the typhoeus and ethon gems into a folder and mounting it in /opt/bitnami/fluentd/plugins

mkdir plugins
cd plugins
gem fetch typhoeus
gem fetch ethon
cd ..
docker run --rm --name fluentd -v $(pwd)/plugins:/opt/bitnami/fluentd/plugins bitnami/fluentd:latest

To be honest I didn't test it properly but if you could confirm it works I will document it

[kvokka]

Hey @fmulero

Thank you for your response. Since i'm using fluentd in k8s in bitnami helm chart, I found useful this option:

    command: ["/bin/sh"]
    args:
      - "-c"
      - |
        /opt/bitnami/scripts/fluentd/setup.sh && \
        gem uninstall -x --ignore-dependencies elasticsearch elasticsearch-api elastic-transport && \
        gem install --no-document 'typhoeus' \
                    'elasticsearch-api:7.17.0' \
                    'elasticsearch-xpack:7.17.0' \
                    'elasticsearch:7.17.0' && \
        /opt/bitnami/scripts/fluentd/entrypoint.sh /opt/bitnami/scripts/fluentd/run.sh

Because occasionally it's required to install an older elasticsearch gem anyway for my setup (the latest version of the gem is not working correctly). Plus, wip on elastic upgrade makes this hack applicable for a short time in my case.

This issue is more about balance and consistency rather than the way how to install the gem in the image. After moving to the latest elastic, the lack of typhoeus gem will be the only cause why I should make a wrapper for the image or hacks as you see above.

[fmulero]

Thanks @kvokka for sharing that solution, that's a great idea if you have to use a particular elasticsearch dependency.

I've opened an internal discussion to allow users add easily new pugins to the container. Unfortunately I can't give you an ETA. Any news will be shared in this case

[kvokka]

Thank you @fmulero !

I'll be happy to get rid of this hack, but the trick is that the official bitnami elasticsearch image does not support version 8.x (issue)

If you do not mind, I can make the PR with simple dependencies management on the fly, like this hack but in initContailer and with a few environment variables, totally optional. It might solve all the issues. I will be happy to implement it if you like the idea

[fmulero]

About the elastic version, we are working on that, we hope to have it soon.

I think you are able to add initContainers and shared volumes in the chart If you think that isn't enough, please feel free to create the PR we will be happy to review it.

[kvokka]

@fmulero Hey! Just wanted to remind you about this idea of adding typhoeus gem to the bundle. Elastic 8 is now live, so it's kinda odd to have the hack mentioned above to be needed.

[fmulero]

Thanks @kvokka, I'll reopen this issue

[fmulero]

We are going to transfer this issue to bitnami/containers

In order to unify the approaches followed in Bitnami containers and Bitnami charts, we are moving some issues in bitnami/bitnami-docker-<container> repositories to bitnami/containers.

Please follow bitnami/containers to keep you updated about the latest bitnami images.

More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html

[carrodher]

Unfortunately, this suggestion was added a year ago and although there is an internal task to evaluate it, it was not planned as something to address in the short/mid term. It's not a technical reason but something related to the capacity since we're a small team.

I totally agree with you that is weird having the oj gem but not others. In the end, this is an objective decision and it will depend on the use case and the different scenarios. The gem that is interesting for you can be irrelevant to another user. That's why we are trying to move (little by little) to a more customizable approach where the assets don't include any customization out of the box but there are methods for users to add the required packages or components.

Being said that, in this case, the decision was to not include typhoeus by default in the container image. At this moment there is a workaround to include it (although it is not the best option) and we will work on improving how the container image can be customized.