s3-sqs-connector
s3-sqs-connector copied to clipboard
Bug Fix to filter uncommitted messages when sorting is disabled and m…
…axFilesPerTrigger is set
Bug/Issue : Incorrect filter condition, leading to no files being given to Spark
When working with the connector code, we had to disable the sorting to gain performance improvement in our application. After disabling sorting, we noticed that the condition to filter the uncommitted messages from the cache is incorrect and will always be false.
Since we want to filter uncommitted messages, the condition should be looking for files that are not committed, similar to the code flow path here: https://github.com/qubole/s3-sqs-connector/blob/master/src/main/scala/org/apache/spark/sql/streaming/sqs/SqsFileCache.scala#L118
@@abhishekd0907 Can you review this change ?