[New Chart][Logstash]

[TheAlgo]

Is your feature request related to a problem? Please describe. A chart for Logstash will be helpful in deploying OpenSearch stacks to Kubernetes

Describe the solution you'd like N/A

Describe alternatives you've considered N/A

Additional context This will help cater many use-cases for log ingestion, analytics etc.

[sastorsl]

I actually got the bitnami-chart to work with opensearch logstash, but not 100%. It could perhaps form a basis for this?

[TheAlgo]

@sastorsl certainly we can start from there.

[michaelharrer]

Are there any updates regarding this chart? @sastorsl is there some documentation about the bitnami solution? A "curated" chart would be really, really helpful.

[verbathim]

I would also really like to have an "official" chart which gets maintained.

[TheAlgo]

+1 on the need of an official chart for Logstash

[sastorsl]

Would really like to see this coming. It does not however seem that I'm able to work on this in the foreseeable future.

/tag helpwanted ...

[tgeek77]

I'm not the most experienced at writing helm charts, but looking at the docker images from opensearch.org and the one used by bitnami, they are quite similar and perhaps it won't be a huge amount of work to replace the opensearch image with the bitnami image. The best part is that the bitname helm chart is released with an Apache license so there shouldn't be any licensing issues. Opensearch Image: docker.io/opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 Btinami Image: docker.io/bitnami/logstash:8.6.2-debian-11-r3

[ahmadabulaban]

Any updates on this issue ?

Really appreciated if we can have an official chart for opensearch logstash

Now the options is to use

https://artifacthub.io/packages/helm/bitnami/logstash (last version 5.3.2 - 2 May, 2023) vs https://artifacthub.io/packages/helm/elastic/logstash (last version 8.5.1 - 16 Nov, 2022)

The bitnami one released with an Apache license, but the file structure of docker image is different. I see that the official logstash chart is the similar one from docker image structure point of view

[sastorsl]

The elastic chart works, if you mount a couple of directories. However, elastic is sunsetting their helm charts, akà handing them over to the community.

    logstashPipeline: {}  # Se ConfigMap / extraVolumes / extraVolumeMountes
    logstashConfig:
      logstash.yml: |
        http.host: 0.0.0.0
        config.reload.automatic: true
        pipeline.ecs_compatibility: "disabled"
        pipeline.batch.size: 1250
        # xpack.monitoring.enabled: false  # Not in use with the opensearch version of logstash
    extraVolumes: |
      - name: logstash-patterns
        configMap:
          name: custom-logstash-patterns
      - name: logstash-conf-d
        configMap:
          name: custom-logstash-conf-d
    extraVolumeMounts: |
      - mountPath: /usr/share/logstash/patterns  # GROK patterns
        name: logstash-patterns
        readOnly: true
      - mountPath: /usr/share/logstash/pipeline/logstash.conf  # Input, filter and output config
        name: logstash-conf-d
        subPath: logstash.conf
        readOnly: true

Guess I'll check out the bitnami charts.

[sastorsl]

Duplicate(?) of #13 ?

[sastorsl]

I was able to successfully convert from the elastic chart to the bitnami chart.

1. logstash.conf is maintained in a ConfigMap
2. GROK Patterns are maintained in a ConfigMap
3. I've set input, filter, and output to null - i.e. input: null
4. And mounted patterns and config as extraVolumeMounts as described above.
5. Mapped helm chart config options "namespace"
6. NB! Override the default bitnami runAsUser and fsGroup (they use uid/gid 1001 instead of 1000)
7. Ensured that I've added the monitoring port to containerPorts to support the Lifecycle config

All in all it works.

Some more input to the licensing and other concerns would of course be welcome.