Got success message from uppy but the file didnot get uploaded to s3

[IndrajaPunna]

We are using aws multipart upload with companion server setup. Around 360 files were uploaded at a time with each file being of size >20MB. When uppy gave a success message, we saved the s3 url reference to our database. We realised that some files are not present in s3 but we got a success message from uppy. There is a mismatch of around 1000 files. Can somebody help on what can be the reason for this. We are using uppy on production. Kindly, let us know if using a paid plugin wont cause an issue of this sort.

[mifi]

S3 used to be eventually consistent, so theoretically after upload success, the objects could still return 404. But it seems like they recently changed this behaviour so that they now guarantee strong consistency: https://www.infoq.com/news/2020/12/aws-s3-strong-consistency/

So this seems to be a bug with uppy or companion then. Is it possible to provide a reproducable case, or instructions for how to reproduce this issue?

[paulalex]

I have experienced something similar I had to handle in my custom API, when sending multipart uploads and completing, uppy, for some unknown reason will include empty objects. In python these get converted into empty dictionaries which then cause S3 API to fail. This is only noticeable when using the the pause functionality and when uppy has to query for currently uploaded parts.

I remove any empty dicts from the parts list now and this solves the problem. Presumably the code which is sending the list of parts to complete a multi part upload in uppy should also remove any empty javascript objects from this array of part objects as they cause intermittent errors when trying to complete the S3 multi part upload.

I noticed similar behaviour in that uppy says it uploaded successfully even though my API returns a 500 error which is visible in the javascript console, no file ends up in S3 (obviously) and so of course my async function doesnt fire to update dynamo DB.

You may see a similar error to this in your logs:

An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well-formed or did not validate against our published schema

[mifi]

Where were files coming from? A remote provider like the Link plugin, Google Drive, Onedrive etc, or from the browser? (My device, Camera etc).

What I'm really wondering about is whether this is problem happens with the /:providerName/get endpoint or with the /s3/multipart/* endpoints.

[Murderlon]

@paulalex @IndrajaPunna could you provide us with more details like what @mifi is asking for?

[Murderlon]

Closing this as it seems to be fixed. Can reopen if needed :)