kinesis-sql
kinesis-sql copied to clipboard
Consuming from multiple Kinesis Streams
Hi, I have a consumer application which needs to connect/consume from multiple Kinesis Streams. However, using this library, I see that we can consume from only a single stream. While trying to read from multiple Kinesis stream I got the following error: Caused by: org.apache.spark.sql.kinesis.shaded.amazonaws.services.kinesis.model.AmazonKinesisException: 1 validation error detected: Value 'Stream_1,Stream_2' at 'streamName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+ (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: f5bf9ad6-471f-17d7-a1e2-f6de108bdde7)
Could you shed some light on how we can mention multiple streams or if that is even possible from the same consumer?
Thanks in advance.
@sap2105 - The library currently supports single stream. We need to make changes to handle multiple streams. Would you be interested in working on it and raising a PR?