Size field of files imported from Facebook and Instagram is null

[netdown]

Hi,

As the title indicates, the size field of the FileObject is null in case of Facebook and Instagram files.

Object dump: { "source": "Instagram", "id": "uppy-instagram/2020/07/29t13/50/39/00000/jpeg-10-1d-1d-1q-1q-1b-1e-image/jpeg-null", "name": "Instagram 2020-07-29T13:50:39+00000.jpeg", "extension": "jpeg", "meta": { "name": "Instagram 2020-07-29T13:50:39+00000.jpeg", "type": "image/jpeg" }, "type": "image/jpeg", "data": { "isFolder": false, "icon": "*****a94bad99bb894995&oe=61A7F2ED", "name": "Instagram 2020-07-29T13:50:39+00000.jpeg", "mimeType": "image/jpeg", "id": "17884245532629029", "size": null, "thumbnail": "*****a94bad99bb894995&oe=61A7F2ED", "requestPath": "17884245532629029", "modifiedDate": "2020-07-29T13:50:39+0000" }, "progress": { "percentage": 0, "bytesUploaded": 0, "bytesTotal": null, "uploadComplete": false, "uploadStarted": null }, "size": null, "isRemote": true, "remote": { "companionUrl": "*****", "url": "*****/instagram/get/17884245532629029", "body": { "fileId": "17884245532629029" }, "providerOptions": { "companionUrl": "*****", "provider": "instagram", "pluginId": "Instagram" }, "providerName": "Instagram" }, "preview": "****fea94bad99bb894995&oe=61A7F2ED" }

Note that sending null metavalues to companion leads to crashes, I believe this also must be fixed.

TypeError: Cannot read property 'name' of null\n    at FormData._getContentDisposition (node_modules/form-data/lib/form_data.js:226:40)\n    at FormData._multiPartHeader (node_modules/form-data/lib/form_data.js:177:33)\n    at FormData.append (node_modules/form-data/lib/form_data.js:70:21)\n    at appendFormValue (node_modules/request/request.js:326:21)\n    at Request.init (node_modules/request/request.js:337:11)\n    at new Request (node_modules/request/request.js:127:8)\n    at request (node_modules/request/index.js:53:10)",

at node_modules/request/index.js:61:12"
at Uploader.uploadMultipart (node_modules/@uppy/companion/lib/server/Uploader.js:446:13)

[Murderlon]

Hi, size is not always returned by the providers so that's is not something we can do much about unfortunately.

Regarding the second error, that seems to be something different. Can you provide a reproducible example?

[netdown]

Just set a custom metadata with null value.

[mifi]

What kind of instagram/facebook files are giving size null? last time i tested instagram/facebook with uppy it did work.

I just tested instagram on https://uppy.io/ and it works. Could you provide details how to reproduce the issue?

[netdown]

I have no idea, I tested it with my personal accounts and each image I have tried had null as size. Actually, Instagram did not work at all above 2.11.1 (it threw a misconfigured provider error), but this null size error happened with all releases, I have tried almost all minor versions from major release 2. Companion is running as a standalone server.

[mifi]

Ok. It's a bit hard to test if we don't have a way to reproduce it, but in the next version of companion v3 (unreleased yet) we did a complete rewrite of the uploader, so it should now support size null. I want to confirm this first. I think I will just edit the code and hardcode it to null to see that it indeed works, since we don't have a way to reproduce it.

[mifi]

Actually when I think about it, https://uppy.io is already running the new unreleased version of companion. could you try with your account there and see if it works?

[netdown]

Checking the list query's response, each file size is null. Note that the import/upload process is not influenced by this, only if you'd like to utilize that size field for sg.

[mifi]

What is "sg"? I'm not sure what is exactly the issue. Are you getting this error TypeError: Cannot read property 'name' of null when uploading files from instagram/facebook? If so, can you also reproduce the crash on uppy.io? If not, then I believe it is fixed in the newest version of companion.

[netdown]

Sg is an abbreviation for something. As I have stated, there are different 2 problems, which linked in my case:

1. size attribute is null for images imported from Facebook/Instagram. This also hapens on uppy.io. This does not affect the basic upload process, I can import images from Facebook/Instagram both on my site and uppy.io. But if I'd like to utilize the size attribute for example to compare expected and actual file sizes, then the problem arises.
2. The Companion TypeError happened when I tried to pass the size attribute as a metadata. As uppy.io does not do this, this is not reproducible on that site. The Companion related problem is that it cannot handle null values in metadata, that is causing this TypeError. You can reproduce it if you manually send null as some metadata.

[mifi]

Thanks for your clarifications.

1. I think the culprit is here: https://github.com/transloadit/uppy/blob/78268d00fd46f4465ca2c2134f067129a8c25232/packages/%40uppy/companion/src/server/provider/facebook/index.js#L162 I just added a console.log with facebook but I believe the problem is that the facebook API doesn't return any size of the item, so that's why. not much we can do about that.
2. What's the use case for setting metadata to null? TypeError sounds correct to me. Or is the problem that the whole companion server crashes?

[netdown]

Of course I was not intentionally setting null values. Companion crashes and the upload process hangs at 50 percent. TypeError is thrown by form-data package, not Companion itself.

[mifi]

Ok im going to try to reproduce this. you are using the XHRUpload plugin, correct?

[netdown]

Yes.

[mifi]

Ok, I just tested locally from main branch, and I can confirm there is an error log [error] 088a8591 uploader.error TypeError: Cannot read properties of null (reading 'name') spit out from companion. However companion did not crash and the /get request did return with 200. uppy.on('complete', (result) => result.failed) did return the file in question, and the uppy UI does report "upload failed" which is correct. So in main this seems to be working as expected. This could be due to some unreleased fixes, so either you can try companion from main, or wait for the next minor release of companion v3. If you are still able to reproduce the crash on main, then please provide some more details on how to reproduce it.

[Murderlon]

@netdown can you still reproduce on latest as @mifi explained?

[Murderlon]

Closing this but we can continue the discussion if needed