SQS Message Parsers (e.g. S3EventNotification)

[ravikancherla]

I am trying to upload a file to s3 and added a event notification pointing to sqs queue. I was using S3EventNotification class to parse it earlier in SDK 1.x but I don't see that class anymore.

[spfink]

We have not yet ported that class over to the v2 SDK. You should be able to continue to use the class from v1 just fine.

[yaitskov]

how about conflicts between transitive dependencies from 1.1 and 2.0?

[jeffalder]

@millems - should we port this class over as-is, or should we modify codegen to emit new shapes that aren't present in the service definition? Are there similar shapes in other services that aren't contained in the service definition?

[millems]

Hmm, I'm not sure how this should be done. I don't mind hand-writing them, but I'm not sure how well we can keep up with changes that might happen in the future. Let me check on our end to see if we want to hand-write these or find some way to generate them.

[jochs]

Has there been any movement on this in the past few months? We have an internal product which has migrated entirely to the 2.x sdk except for parsing these event notifications and I'd like to clean up the dependency if possible

[joain946]

Is this issue on the roadmap? When is this planned to be released? To be able to listen to an S3 event it means that three jars from the old SDK will be needed (s3, core and kms). Maybe it is possible to exclude some of them manually. If not then it means that this will add almost 3 MB to my Lambda which is not optimal.

Since the class (S3Event) has already been created and that there are already other Json objects that has been created for V2 it can't be that much work to add this as well.

As far as I can see this is the only class that this needs to be corrected for in all the 23 classes in the "aws-lambda-java-events" jar file. So it would really help if this could be prioritized.

[millems]

It's on a roadmap, but it's far enough down the list that there's no date assigned. I think this will be a tricky one, since it's not clear who on the AWS side would be best to own this. We're not very well equipped to keep it up to date.

That said, the SDK owns this for V1, so it's on us to figure out the next steps. It might be easier to prioritize if we start by trying to figure out who should own this. We'll start with that. That said, I can't guarantee any dates for that, so it's still "planned, but not at the top of the backlog".

[joain946]

While on the topic of minimizing dependencies. Is there a reason why three of the event classes uses this: "import org.joda.time.DateTime;". All other v2 Api:s like S3 and KMS uses "java.time.Instant". The Joda time dependency adds about 600 kB which would have been nice to skip.

I understand that this can't be changed without breaking existing code but maybe new classes can be created so that developers can choose to manually exclude Joda time.

[millems]

Anything that is owned by the AWS SDK for Java team going forward will not use Joda time. You can provide this feedback to the Lambda Runtime team on their repository: https://github.com/aws/aws-lambda-java-libs/tree/master/aws-lambda-java-events

[raseals]

Any update on a port of S3EventNotification from aws-java-sdk v1 -> v2? We have a project that was forced to migrate to using v2 because v1 will not work in AWS C2S (gov classified) because of Region names not in the enum. Would be nice to have v2 updated to support S3EventNotifications when an S3 bucket gets a new object ... as our ingest feed sends an S3EventNotification to an SQS Queue. We are still using v1 to handle parsing the SQS message and parsing it to an S3EventNotification (v1).

[joain946]

The "com.amazonaws.services.lambda.runtime.events.S3Event" class is already available in the "aws-lambda-java-events" artifact. I have also excluded "apache-client" and "netty-nio-client" from that artifact which works for my purpose.

"com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification.S3EventNotificationRecord" might be the more specific class you are asking for.

[mleprince]

any update on this ? I use SQS to process S3 events, I would not like to import aws-lambda-java-events only for this purpose.

[bluenautilus2]

Please please fix this. Adding both the v1 and v2 libraries to our lambdas causes the jars to be bigger, and costs us more memory in the long run.

[AlyHdr]

Same issue here been looking for two days for a solution and I ended up over this issue !! Having both versions in my project can cause problems.

I'm trying to parse an S3EventNotification in a lambda input, we had this mapper in V1 but there is no equivalent in V2.

[sandria-s]

Any update or workaround? We're also trying to parse the S3EventNotification from SQSEvent Body

[RichardFourie]

Would also like to have this fixed since we are importing the v1 libs only for this single class.

[debora-ito]

As a reminder, add a 👍 in the original description if you want us to support this feature. The number of reactions helps us when we prioritize features.

[malets12]

Found a solution without direct usage of SDK V1, but with this dependencies:

implementation 'com.amazonaws:aws-lambda-java-serialization:1.0.0'
implementation 'com.amazonaws:aws-lambda-java-events:3.11.0'

In my case S3EventNotification.fromJson(String.class) was replaced with LambdaEventSerializers.serializerFor(Clazz<T>, Thread.currentThread().getContextClassLoader()).fromJson(Sting.class) (where Clazz<T> is S3EventNotification.class or S3Event.class, etc.) during migration. However, hope there will be a better way soon.

[jgangemi]

i just ripped the class out of the v1 sdk and added it to my project. the only issue i had was converting the event timetamp using LocalDateTime. i also don't need that field for my use, so i'm fine w/o it (i just commented out the code) here it is for anyone else who doesn't want all the extra library bloat.

import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import software.amazon.awssdk.utils.http.SdkHttpUtils;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;


// TOOD: remove when part of aws sdk
public class S3EventNotification
{
    private final List<S3EventNotificationRecord> records;

    @JsonCreator
    public S3EventNotification(
            @JsonProperty(value = "Records") List<S3EventNotificationRecord> records)
    {
        this.records = records;
    }

    @JsonProperty(value = "Records")
    public List<S3EventNotificationRecord> getRecords()
    {
        return records;
    }

    public static class UserIdentityEntity
    {
        private final String principalId;

        @JsonCreator
        public UserIdentityEntity(
                @JsonProperty(value = "principalId") String principalId)
        {
            this.principalId = principalId;
        }

        public String getPrincipalId()
        {
            return principalId;
        }
    }

    public static class S3BucketEntity
    {
        private final String name;
        private final UserIdentityEntity ownerIdentity;
        private final String arn;

        @JsonCreator
        public S3BucketEntity(
                @JsonProperty(value = "name") String name,
                @JsonProperty(value = "ownerIdentity") UserIdentityEntity ownerIdentity,
                @JsonProperty(value = "arn") String arn)
        {
            this.name = name;
            this.ownerIdentity = ownerIdentity;
            this.arn = arn;
        }

        public String getArn()
        {
            return arn;
        }

        public String getName()
        {
            return name;
        }

        public UserIdentityEntity getOwnerIdentity()
        {
            return ownerIdentity;
        }
    }

    public static class S3ObjectEntity
    {
        private final String key;
        private final Long size;
        private final String eTag;
        private final String versionId;
        private final String sequencer;

        @Deprecated
        public S3ObjectEntity(
                String key,
                Integer size,
                String eTag,
                String versionId)
        {
            this.key = key;
            this.size = size == null ? null : size.longValue();
            this.eTag = eTag;
            this.versionId = versionId;
            this.sequencer = null;
        }

        @Deprecated
        public S3ObjectEntity(
                String key,
                Long size,
                String eTag,
                String versionId)
        {
            this(key, size, eTag, versionId, null);
        }

        @JsonCreator
        public S3ObjectEntity(
                @JsonProperty(value = "key") String key,
                @JsonProperty(value = "size") Long size,
                @JsonProperty(value = "eTag") String eTag,
                @JsonProperty(value = "versionId") String versionId,
                @JsonProperty(value = "sequencer") String sequencer)
        {
            this.key = key;
            this.size = size;
            this.eTag = eTag;
            this.versionId = versionId;
            this.sequencer = sequencer;
        }

        public String getKey()
        {
            return key;
        }

        public String getSequencer()
        {
            return sequencer;
        }

        @Deprecated
        @JsonIgnore
        public Integer getSize()
        {
            return size == null ? null : size.intValue();
        }

        @JsonProperty(value = "size")
        public Long getSizeAsLong()
        {
            return size;
        }

        public String getUrlDecodedKey()
        {
            return SdkHttpUtils.urlDecode(getKey());
        }

        public String getVersionId()
        {
            return versionId;
        }

        public String geteTag()
        {
            return eTag;
        }
    }

    public static class S3Entity
    {
        private final String configurationId;
        private final S3BucketEntity bucket;
        private final S3ObjectEntity object;
        private final String s3SchemaVersion;

        @JsonCreator
        public S3Entity(
                @JsonProperty(value = "configurationId") String configurationId,
                @JsonProperty(value = "bucket") S3BucketEntity bucket,
                @JsonProperty(value = "object") S3ObjectEntity object,
                @JsonProperty(value = "s3SchemaVersion") String s3SchemaVersion)
        {
            this.configurationId = configurationId;
            this.bucket = bucket;
            this.object = object;
            this.s3SchemaVersion = s3SchemaVersion;
        }

        public S3BucketEntity getBucket()
        {
            return bucket;
        }

        public String getConfigurationId()
        {
            return configurationId;
        }

        public S3ObjectEntity getObject()
        {
            return object;
        }

        public String getS3SchemaVersion()
        {
            return s3SchemaVersion;
        }
    }

    public static class RequestParametersEntity
    {
        private final String sourceIPAddress;

        @JsonCreator
        public RequestParametersEntity(
                @JsonProperty(value = "sourceIPAddress") String sourceIPAddress)
        {
            this.sourceIPAddress = sourceIPAddress;
        }

        public String getSourceIPAddress()
        {
            return sourceIPAddress;
        }
    }

    public static class ResponseElementsEntity
    {
        private final String xAmzId2;
        private final String xAmzRequestId;

        @JsonCreator
        public ResponseElementsEntity(
                @JsonProperty(value = "x-amz-id-2") String xAmzId2,
                @JsonProperty(value = "x-amz-request-id") String xAmzRequestId)
        {
            this.xAmzId2 = xAmzId2;
            this.xAmzRequestId = xAmzRequestId;
        }

        @JsonProperty("x-amz-id-2")
        public String getxAmzId2()
        {
            return xAmzId2;
        }

        @JsonProperty("x-amz-request-id")
        public String getxAmzRequestId()
        {
            return xAmzRequestId;
        }
    }

    public static class S3EventNotificationRecord
    {
        private final String awsRegion;
        private final String eventName;
        private final String eventSource;
        private final String eventVersion;
        private final RequestParametersEntity requestParameters;
        private final ResponseElementsEntity responseElements;
        private final S3Entity s3;
        private final UserIdentityEntity userIdentity;
//        private LocalDateTime eventTime;

        @JsonCreator
        public S3EventNotificationRecord(
                @JsonProperty(value = "awsRegion") String awsRegion,
                @JsonProperty(value = "eventName") String eventName,
                @JsonProperty(value = "eventSource") String eventSource,
                @JsonProperty(value = "eventTime") String eventTime,
                @JsonProperty(value = "eventVersion") String eventVersion,
                @JsonProperty(value = "requestParameters") RequestParametersEntity requestParameters,
                @JsonProperty(value = "responseElements") ResponseElementsEntity responseElements,
                @JsonProperty(value = "s3") S3Entity s3,
                @JsonProperty(value = "userIdentity") UserIdentityEntity userIdentity)
        {
            this.awsRegion = awsRegion;
            this.eventName = eventName;
            this.eventSource = eventSource;

//            if (eventTime != null)
//            {
//                this.eventTime = LocalDateTime.from(DateTimeFormatter.ISO_INSTANT.parse(eventTime));
//            }

            this.eventVersion = eventVersion;
            this.requestParameters = requestParameters;
            this.responseElements = responseElements;
            this.s3 = s3;
            this.userIdentity = userIdentity;
        }

        public String getAwsRegion()
        {
            return awsRegion;
        }

        public String getEventName()
        {
            return eventName;
        }

        public String getEventSource()
        {
            return eventSource;
        }

//        public LocalDateTime getEventTime()
//        {
//            return eventTime;
//        }

        public String getEventVersion()
        {
            return eventVersion;
        }

        public RequestParametersEntity getRequestParameters()
        {
            return requestParameters;
        }

        public ResponseElementsEntity getResponseElements()
        {
            return responseElements;
        }

        public S3Entity getS3()
        {
            return s3;
        }

        public UserIdentityEntity getUserIdentity()
        {
            return userIdentity;
        }
    }
}

[edelauna]

I'm in the process of migrating from Spring Boot 2.7 to 3, and this thread was very helpful for an SQS migration. However what was previously done automatically via messageConverter, I needed to craft a custom deserializer/wrapper to now handle. I'm wondering if I've missed something?

My pom file

<dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-starter-sqs</artifactId>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>${aws.sts.version}</version>
</dependency>

SqsListener

@SqsListener(
    value = ["\${spring.cloud.aws.topic.name}"],
)
fun ingressListener(message: S3EventNotificationWrapper) {
    message.body.records
        .map { it.s3 }
        .forEach {
             // ...
         }
}

@JsonIgnoreProperties(ignoreUnknown = true)
class S3EventNotificationWrapper(
    @JsonProperty("Message")
    @JsonDeserialize( using = InnerJsonDeserializer::class)
    val body: com.amazonaws.services.s3.event.S3EventNotification
)

class InnerJsonDeserializer: JsonDeserializer<com.amazonaws.services.s3.event.S3EventNotification>() {
    override fun deserialize(p0: JsonParser, p1: DeserializationContext): com.amazonaws.services.s3.event.S3EventNotification {
        val s = p0.valueAsString
        val jsonParser2 = p0.codec.factory.createParser(s)

        return jsonParser2.readValueAs(com.amazonaws.services.s3.event.S3EventNotification::class.java)
    }
}

[noeljohnk007]

As a temp solution, use both s3 aws-java-sdk v1 & v2. v1 for parsing the s3Event & v2 for everything else.

import com.amazonaws.services.s3.event.S3EventNotification;
S3EventNotification s3Event = S3EventNotification.parseJson(sns.getMessage());

Here the S3EventNotification comes from, implementation group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: aws-java-sdk_v1

Obviously adding both v1 & v2 libraries makes the jars to be bigger.

[BKG4211]

This issue was opened in April 2019. In October 2023 (4.5 YEARS later) the best that Amazon has come up with is to use v1 & v2 side by side as a "temp" solution. Sorry, 4+ years is not temporary.

This appears to be a hot potato that no one wants to own. The tricky part isn't technical, it is ownership. So your customers end up having to work around AWS' lack of will to take ownership.

This should be escalated to your executive team to see what they think of this train wreck. A certain other colorful cloud provider isn't perfect, but they are looking better by the day.

[pechenoha]

Hello from 2024. It's been almost 5 years of successfully ignoring the problem. Bravo, Amazon! 🎉

[vsarris]

Hello! It's a pity just 3-4 classes weren't transferred into the sdk v2. I hope at least these events won't change their definition and even v1 classes won't work

[peterson-dc]

This is a blocker for me in moving completely over to the v2 SDK. While I can make my own object representation of the event, I would much rather have AWS manage this object in case of any changes to the contract. Please, we really need this fixed soon since the v1 SDK is EOL.

[BKG4211]

This is a blocker for me in moving completely over to the v2 SDK. While I can make my own object representation of the event, I would much rather have AWS manage this object in case of any changes to the contract. Please, we really need this fixed soon since the v1 SDK is EOL.

Yeah, you would think that a company the size of AWS would understand and support established governance practices. You know, like not putting their customers into a situation of facing an audit with unsupported libraries/software version in their tech stack.

[zoewangg]

Hey all, we are pleased to announce that we have released S3 event notification in 2.25.11 https://central.sonatype.com/artifact/software.amazon.awssdk/s3-event-notifications

Check out our README to get started https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/s3-event-notifications As always, feedback is welcome! I'll go ahead and close this issue, but feel free to open new issues if you have any questions/feedback.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.