S3 object keys are not Url decoded in S3Events received Lambda via S3Notification

[ElectricVampire]

Describe the bug

If I upload an Object in S3 bucket where file name has space in it. Eg. "NEW FILE.txt", The object URL becomes - https://myBucket/NEW+FILE.txt which is correct.

Now When I received an event in my lambda via S3Notification. Property -> S3.Object.Key has "NEW+FILE.txt"

Key value should be Url decoded as the real key of this object is "NEW FILE.txt".

Expected Behavior

Key value of object should be "NEW FILE.txt".

Current Behavior

Key value of object should be "NEW+FILE.txt".

Reproduction Steps

• Create a bucket
• Create a lambda
• Add a S3 trigger in lambda
• Upload a object named "NEW FILE.txt" in S3 bucket
• Verify that key name in bucket is "NEW FILE.txt"
• Once you uploaded the object, event will be triggered and will execute lambda
• Check S3.Object.Key property of S3Event object. It shows key value as "NEW+FILE.txt" while it should be "NEW FILE.txt"

Possible Solution

I think while serialization of S3Event, we can just do WebUtility.UrlDecode(key) which will santize the key from encoded chars which were added during url creation.

Additional Information/Context

We all know that having space is filename is not ideal but we depend on our upstream for this filename and they just can't change it.

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.9.2

Targeted .NET Platform

.NET CORE 3.1

Operating System and version

AmazonLinux

[ashishdhingra]

@ElectricVampire Assuming you are using S3Event class in your Lambda, the change you are proposing could be a breaking change for existing customers. S3Event class inherits from Amazon.S3.Util. S3EventNotification from AWSSDK.S3 package. Changing any default behavior (for example changing setter for S3ObjectEntity.Key property to automatically URL Decode the value) could potentially break existing customers.

Ideally if we are consuming the class directly, these types of changes are controlled by a flag (preferably in class constructor). It would also be desirable not to introduce any dependency on additional packages.

Kindly note that the event classes in this repository are normal POCO classes and are used to deserialize JSON received from Lambda event to a .NET object. So these scenarios could potentially be best handled by the end application (Lambda handler in this case). (I'm not ruling out that more flexibility is always desirable)

Thanks, Ashish

[ElectricVampire]

@ashishdhingra "Changing any default behavior (for example changing setter for S3ObjectEntity.Key property to automatically URL Decode the value) could potentially break existing customers."

If someone has a key which contains characters with charset out side of ASCII, S3 event object key will have them encoded while S3 lib its self support other charset as well. Which means that all of the consumers of S3 event object must do url decoding...Otherwise this is a bug in their code.

Lets take very simple use case of dumping small files in DB via lambda when some other process upload them in S3. You get an S3 event, you will use S3 lib where you will pass the key in that S3 event to stream/download the file and you discover a issue in prod with a corner case where file name has non-ASCII chars.

This should be fixed at level where S3Notification is getting generated, so this get fixed in all lang/FW, so everyone don't have to put the same changes in.

Its a bug which requires breaking changes to fix it and fixing it will fix a bug in consumers code which they may don't even know exist. I don't think even changing the the value in existing property will impact anyone as url encoded keys are useless for consumers and they must decode it somewhere in their code flow. If AWS lib decodes it, decoded value will just flow through their existing system.

If we are really really want to be careful, Maybe another property for Key could be introduced such UrlDecodedKey or something like it.

[ashishdhingra]

Need to check the behavior in AWS SDK for .NET and make it consistent here.

[umaskr]

Just a suggestion, Can we just add one property in S3ObjectEntity class which return decoded key? public string DecodedKey => System.Net.WebUtility.UrlDecode(Key);