Deprecate 'from_sns'

[ashleyprimo]

Context

This PR is (at-least initially-hence draft) intended to be a discussion, as I do not fully understand why from_sns is needed, as far as I can tell the payload format when S3->SNS->SQS architecture is deployed, the payload resembles the following:

{
    "Records": [
      {
        "eventVersion": "2.1",
        "eventSource": "aws:s3",
        "awsRegion": "obfuscated",
        "eventTime": "2024-01-01T00:00:00.000Z",
        "eventName": "ObjectCreated:Put",
        "userIdentity": {
          "principalId": "AWS:obfuscated"
        },
        "requestParameters": {
          "sourceIPAddress": "obfuscated"
        },
        "responseElements": {
          "x-amz-request-id": "obfuscated",
          "x-amz-id-2": "obfuscated"
        },
        "s3": {
          "s3SchemaVersion": "1.0",
          "configurationId": "obfuscated",
          "bucket": {
            "name": "obfuscated",
            "ownerIdentity": {
              "principalId": "obfuscated"
            },
            "arn": "arn:aws:s3:obfuscated"
          },
          "object": {
            "key": "obfuscated",
            "size": 0,
            "eTag": "obfuscated",
            "versionId": "obfuscated",
            "sequencer": "obfuscated"
          }
        }
      }
    ]
  }

Which is inline with https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html

So, attempting to payload = JSON.parse(payload['Message']) if @from_sns will fail as of course, the JSON structure does not match that expectation.

There are number of issues I have seen, related to this:

• https://github.com/cherweg/logstash-input-s3-sns-sqs/issues/54
• https://github.com/cherweg/logstash-input-s3-sns-sqs/issues/57
• https://github.com/cherweg/logstash-input-s3-sns-sqs/issues/80

Why removal?

I do not currently understand how this payload would ever be nested within payload['Message']; though if there is a case where this would be the case - we should improve documentation to reflect this, and probably the naming of the variable.

Currently, from_sns needs to equal false even when the data does originate from SNS - contrary to the comment in code Whether the event is processed though an SNS to SQS. (S3>SNS>SQS = true |S3>SQS=false) (and inference made in documentation)

@cherweg maybe you will know if I am missing something here?

[christianherweg0807]

At time of writing the code SNS did not support raw message body delivery.

(https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html)

So a Message delivered from S3-> SNS to SQS had an additional Header. The Original message was in payload['Message'].

Thank you for your idea…

[ashleyprimo]

At time of writing the code SNS did not support raw message body delivery.

(https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html)

So a Message delivered from S3-> SNS to SQS had an additional Header. The Original message was in payload['Message'].

Thank you for your idea…

I see, that makes a lot of sense. I did spend some time searching to try understand, knew there must've been a reason why you added it. 😁

I guess if it's possible that someone might turn this feature off voluntarily (for whatever) reason we can add a special flag to support that use case.

Maybe sns_raw_payload_delivery_disabled with a default set to false?

Happy to alter the PR to add this.

[ashleyprimo]

How is this latest commit @christianherweg0807

[ashleyprimo]

Bump @cherweg / @christianherweg0807 - would be great to get this, or a derivative solution in mainstream to make configuration nicer. 🙂

[cherweg]

Sorry, I’ll be back at the keyboard in 2 Weeks.Von meinem iPhone gesendetAm 01.03.2024 um 08:59 schrieb Ashley Primo @.***>: How is this latest commit @christianherweg0807

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>