numRecords cannot be negative exception

[amolahinge]

hi, We are managing offsets externally in HBase. We haven't enabled spark checkpointing. So everytime we want to start job, we are providing offsets, which are correct when cross checked with Kafka.

Only sometimes we get this exception "numRecords cannot be negative" Is it the problem cause of property "spark.streaming.kafka.consumer.cache.enabled" set to true inside compute method of DirectKafkaInputDStream class or checkpointing which is done internally by spark? or some other issue.

lib: spark-streaming-kafka-0-10_2.11

Thanks in advance.

[koeninger]

Can you share a stacktrace? On Wed, Oct 10, 2018 at 5:16 AM amolahinge [email protected] wrote:

hi, We are managing offsets externally in HBase. We haven't enabled spark checkpointing. So everytime we want to start job, we are providing offsets, which are correct when cross checked with Kafka.

Only sometimes we get this exception "numRecords cannot be negative" Is it the problem cause of property "spark.streaming.kafka.consumer.cache.enabled" set to true inside compute method or checkpointing which is done internally by spark? or some other issue.

lib: spark-streaming-kafka-0-10_2.11

Thanks in advance.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.