kinesis-sql icon indicating copy to clipboard operation
kinesis-sql copied to clipboard

Consuming from multiple Kinesis Streams

Open sap2105 opened this issue 6 years ago • 1 comments

Hi, I have a consumer application which needs to connect/consume from multiple Kinesis Streams. However, using this library, I see that we can consume from only a single stream. While trying to read from multiple Kinesis stream I got the following error: Caused by: org.apache.spark.sql.kinesis.shaded.amazonaws.services.kinesis.model.AmazonKinesisException: 1 validation error detected: Value 'Stream_1,Stream_2' at 'streamName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+ (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: f5bf9ad6-471f-17d7-a1e2-f6de108bdde7)

Could you shed some light on how we can mention multiple streams or if that is even possible from the same consumer?

Thanks in advance.

sap2105 avatar Nov 08 '18 19:11 sap2105

@sap2105 - The library currently supports single stream. We need to make changes to handle multiple streams. Would you be interested in working on it and raising a PR?

itsvikramagr avatar Nov 09 '18 07:11 itsvikramagr