Upload files to Cumulocity as multipart/form-data

[albinsuresh]

Is your feature improvement request related to a problem? Please describe.

The c8y_http_proxy::actor.rs::upload_file() API currently reads the file contents into memory before passing it into the body of the request. This can be problematic if the file size is huge and hence must be avoided.

Describe the solution you'd like

C8y supports file uploads as multipart/form-data as documented here. We should leverage that in the c8y_http_proxy without having to read the entire file contents into memory.

[reubenmiller]

Technically you don't need to send a request as a multipart/form-data to be memory efficient, as it is all about streaming the read of the file when the request is being written to.

[jarhodes314]

Technically you don't need to send a request as a multipart/form-data to be memory efficient, as it is all about streaming the read of the file when the request is being written to.

One thing that generally needs considering when streaming files without multipart is making sure Content-Length is set and handled appropriately at the other end (because for the server receiving the file, there is no distinction between "the client stopped sending data because the connection dropped/they crashed" and "the client stopped sending data because they've sent the entire file"). We definitely need to set it if we send the response without multipart, and need to ensure that Cumulocity pays attention to this.

I don't know how Cumulocity handles partial uploads, but I do know that if you upload a file that is above the binary size limit for the tenant, Cumulocity will respond with a 200 status code and silently truncate the file. It was a while ago when thick edge bumped into this and asked if core could fix it, but I think as usual there was no appetite for correcting the API, so I think this bug still exists (although it's much less likely to be a problem for thin-edge given the chance of a >2GiB upload is probably slim). It's easy to test that issue still exists, you can do it by uploading a large file to e.g. the firmware repository via the UI and check that the upload is rejected.