graylog2-server
graylog2-server copied to clipboard
Graylog2
Allow two Graylog clusters to read from same Kinesis steam
What?
Allow two different Graylog clusters to use the AWS Kinesis CloudWatch Input to read from the same Kinesis stream simultaneously.
Why?
I am starting a project to migrate from Graylog V3.x to Graylog 5.x clusters. My plan is to run both clusters in parallel so users can use both while they transition all of their Alerts/Dashboards/etc from old to new.
I would like the ability to have an Input on both clusters that read from the same Kinesis stream. As it sits today, it appears that I would have to disable the Input on the old cluster in order for the new cluster to see new messages on the stream as they both use the same DynamoDB table to track where they are in reading data in the stream.
For example, the stream name is eventflow and the DynamoDB table name (for both) is graylog-aws-plugin-eventflow. Seems like the solution would be to make the DynamoDB table name unique for each Graylog cluster?
Please advise if there is already a way to do this and I have overlooked how to do so.
Your Environment
- Graylog Version: 5.2.3
- OpenSearch Version: 2.11.1
- MongoDB Version: 6.0.12
- Operating System: Amazon Linux 2
- Browser version: NA
Yes, you are correct. For Kinesis to support multiple consumers the application name used in the client needs to be different, which right now it isn't.
I suppose adding the cluster id to the name will solve the issue. For reference https://github.com/Graylog2/graylog2-server/blob/3afdedfc60015e9dda8a73a00c78f3844ddc0818/graylog2-server/src/main/java/org/graylog/integrations/aws/transports/KinesisConsumer.java#L129 is using a fixed name, and we need to make this unique per cluster.
@kroepke , any updates on this? Fixed in one of the 5.2.x releases or in the new 6.0.x?
Hi @stepkirk, this feature has not been completed as of yet. It is in a short list to be completed but has run into higher priorities that have delayed our ability to get to it. The Product Owner will definitely update the status when this does move forward.
