kinesis-sql

kinesis-sql copied to clipboard

Data not receiving from Kinesis

8

Hi, I am doing a POC with kinesis and am using this connector and am hoping to use this in production. When "TRIM_HORIZON" is used in a newly created stream,...

success-m

startingPosition AT_TIMESTAMP iteratorOption NumberFormatException

9

Excellent addition for reading from stream at specific positions per [#78](https://github.com/qubole/kinesis-sql/pull/78) However, I'm having trouble using the "AT_TIMESTAMP" option to read from shards at specific timestamps. The shardInfo object I'm...

adamlbailey

The PR closes https://github.com/qubole/kinesis-sql/issues/111

akshayar

I was trying to connect to Kinesis stream from a Spark streaming job running on EME on EKS on a Fargate container. The code snippet I tried is following -...

akshayar

Fix listShards to only specify nextToken

2

This fixes two bugs: 1. If `listShards` returns multiple pages, then we run into an infinite loop with `while (!nextToken.isEmpty)`, because `nextToken` will never be set back to `null` because:...

Anthuang

What may be causing and how to work around StreamingQueryException: Gave up after 3 retries while fetching MetaData ?

2

Spark 3.1.1, running in AWS EMR 6.3.0, python 3.7.2 I'm getting the following error: ``` File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/streaming.py", line 101, in awaitTermination File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line...

dgoldenberg-audiomack

java.lang.IllegalStateException: /checkpoint/round/sources/0/shard-commit/1 does not exist

27

Hey, I am using Kinesis-sql repo 2.4.0 branch and when i am trying to write to checkpoint to S3 i am facing this issue. Initially i thought this might be...

HariprasadAllaka1612

Kinesis AWS client v2 has a support to consume the data through fanout consumer to achieve consistent consumption of 2mb per second per partition. does this support the kinesis fanout...

itsmesrds

Kinesis Enhanced fan-out consumer ( using consumer ARN)

3

I am able to use consume the Kinesis stream using this jar as a normal consumer. When i updated the user account to Enhanced fan out consumer, i am unable...

senthilkumarkk

How do I ensure checkpointing for kafka streams?

2

Hi, I have a stream and the current code is repeating the processing of data in streams when restarted. I wanted to make sure that there is some checkpoint written...

Vikas-kum