s3-sqs-connector icon indicating copy to clipboard operation
s3-sqs-connector copied to clipboard

Bug Fix to filter uncommitted messages when sorting is disabled and m…

Open Aishwarya2203 opened this issue 10 months ago • 1 comments

…axFilesPerTrigger is set

Bug/Issue : Incorrect filter condition, leading to no files being given to Spark

When working with the connector code, we had to disable the sorting to gain performance improvement in our application. After disabling sorting, we noticed that the condition to filter the uncommitted messages from the cache is incorrect and will always be false.

Since we want to filter uncommitted messages, the condition should be looking for files that are not committed, similar to the code flow path here: https://github.com/qubole/s3-sqs-connector/blob/master/src/main/scala/org/apache/spark/sql/streaming/sqs/SqsFileCache.scala#L118

Aishwarya2203 avatar Apr 20 '24 09:04 Aishwarya2203

@@abhishekd0907 Can you review this change ?

Aishwarya2203 avatar Apr 20 '24 09:04 Aishwarya2203