s3-sqs-connector

s3-sqs-connector copied to clipboard

Bug Fix to filter uncommitted messages when sorting is disabled and m…

1

…axFilesPerTrigger is set Bug/Issue : Incorrect filter condition, leading to no files being given to Spark When working with the connector code, we had to disable the sorting to gain...

Aishwarya2203

arunravimv

Maximum Throughput Limited to 1 Message per Second

1

We recently faced an issue with the maximum throughput of the s3-sqs connector. Apparently, in the fastest possible case, the fetching of a message from the target SQS queue can...

arashta

Is it Work with spark 3?

5

Hello, Thank you for this great connector. Is it work with spark 3? This connector is only supported by this repository? I found in apache bahir repository too.

rubenssoto

removing unnecessary lower limit on fetch interval

2

removing the limit on sqsFetchInterval

sujeet-srivastava

Fix max throughput issue

3

This pull request fixes the issue report in https://github.com/qubole/s3-sqs-connector/issues/9.

tavakara

SqsClient: Unexpected error while parsing SQS message next on empty iterator

1

Error while parsing SQS message The below data has been sent to SQS queue via python boto3 library: ``` data = {'ticket_id': '3000613'} # Send message to SQS queue response...

kartthikn

### Background S3-SQS source doesn't support reading partition columns from the S3 bucket. As a result, the dataset formed using S3-SQS source doesn't contain the partition columns leading to issue...

abhishekd0907

S3-SQS source does not populate partition columns in the dataframne

6

Hi, I are using this "s3-sqs" connector with spark structured streaming and deltalake to process incoming data in partitioned s3 buckets. The problem I are facing is with "s3-sqs" source...

DipeshV