At least once delivery vs. delete_on_success

[maederm]

Hi

We're facing the issue that for some S3 files we get the message from the queue multiple times. That would be explainable as the guarantee of the queue is at least once delivery. As we are using the delete_on_success option for the second recipient of the message the file isn't available and it give the error Unable to download file. Requeuing the message. This error is then reappearing every few minutes.

If we were to disable the setting delete_on_success we would ingest the same message multiple times.

Is there a way to tweak the behavior? We could delete the s3 data in another way and handle duplicates with the fingerprint filter, but I hope there would be a nicer solution.

Best, maederm

[HuJake]

@maederm

I would also like to consult this question.

Error log Unable to download file. Requeuing the message {:error=>#<Aws::S3::Errors::NoSuchKey: The specified key does not exist.>