kinesis-sql
kinesis-sql copied to clipboard
qubole
Kinesis Connector for Structured Streaming
Hi, Creating test for this is not really straightforward for me, but this would be a nice feature to have.
The current Kinesis sink only sends one record into Kinesis stream each time, the writing speed is very slow. With the changes in this PR, we could bundle all records...
Hi @itsvikramagr I'm using qubole connector for reading from amazonkinesis streams in spark structured streaming mode. If i change the log level from INFO to DEBUG in log4j.properties, i see...
Hi @itsvikramagr, Thanks again for writing this library and providing constant support to community. Our spark structure streaming code doesn't need to be stateful. We are ok with cluster or...
Hello, recently I was using your libraty for reading/writing from/to kinesis streams. While testing code on local aws instance (https://github.com/localstack/localstack) I had some troubles with writing to kinesis stream. Producer...
Hi guys, Does the library support multiple sinks? I am using two sinks (KinesisSink & FileSink) in the same application. Both sinks use their own checkpoint locations. 1. Read a...
Hi, does the library support proxy settings when connecting to Kinesis public endpoint? My spark application runs within a VPC with a proxy configured to reach AWS public endpoints. Thank...
Hi, I have a consumer application which needs to connect/consume from multiple Kinesis Streams. However, using this library, I see that we can consume from only a single stream. While...
Current work is developed against SPARK 2.2.X. There are new data-source APIs in SPARK-2.3.0. We need to migrate to those APIs
