Google Cloud Storage issues

[nfriedly]

Hi, I'm trying to use tus-node-server & tus-js-client for resumable uploads to Google Gloud Storage,

First up, some context: as I understand it, the chunkSize must be set to lower than the file size in order for it to actually be "resumable", otherwise it just starts over at the beginning after any pause/interruption. So the default size of Infinity on tus-js-client seems like a poor choice given that resumability is the headline feature. But, as far as I can tell, it's the only setting that actually works, at least for Google Cloud Storage. Anything less than the size of the uploaded file hits multiple issues:

1. 409 Upload-Offset conflict. GCS appears to take some time to process uploaded data and update the file size. So, when the second chunk begins, GCS still reports a file size of 0, and then tus-node-server rejects the second chunk because it's starting offset doesn't match the erroneous file size that GCS reports.

   I'm not sure what a good way to solve this is. Disabling the offset check works, but then you're not following the protocol. Adding a 15-second delay also works, but is slow and stupid and brittle.

2. With a hack in place to get around the above, the third chunk starts in the wrong position. This is because, I think, the 0 should be offset here: https://github.com/tus/tus-node-server/blob/master/lib/stores/GCSDataStore.js#L151

   This seems like a straightforward logic error, but the only thing that gives me pause is that it also starts at 0 in https://github.com/tus/tus-node-server/blob/master/lib/stores/FileStore.js#L111 and that one seems to works correctly. So maybe I'm misunderstanding something.

[lostpebble]

Yea this is making Tus basically unusable with Cloud Functions and storage triggers. In fact, I think the nature of Tus itself probably makes it unusable (the chunking nature)... Since it will trigger the function multiple times during the upload (the behaviour I'm currently seeing).

[nfriedly]

FWIW, I figured out where Google's official library for this is: it's part of the Firebase SDK: https://firebase.google.com/docs/storage/web/start

I switched to that and things started working properly.

[Acconut]

Sorry to hear that you were having issue with this.

First up, some context: as I understand it, the chunkSize must be set to lower than the file size in order for it to actually be "resumable", otherwise it just starts over at the beginning after any pause/interruption.

This is not the case, tus-js-client should be able to upload and resume files with any specified chunkSize, whether it be an integer or Infinity. We successfully tested it, so I assume that this was an issue for the tus-node-server data storage for GCS. If I recall correctly, there was once a bug fix for an issue similar to yours but I am not sure. Maybe @bhstahl could provide more insights.

Yea this is making Tus basically unusable with Cloud Functions and storage triggers. In fact, I think the nature of Tus itself probably makes it unusable (the chunking nature)... Since it will trigger the function multiple times during the upload (the behaviour I'm currently seeing).

I am not very familiar with Cloud Functions but you may be able to check at the beginning of your invocation whether the upload is finished yet and only continue if this is the case. The function will still be invoked multiple times during the upload but will only execute its body once.

[chrisdostert]

FWIW, https://github.com/tus/tus-node-server/blob/master/lib/stores/GCSDataStore.js#L151 is definitely wrong as pointed out by @nfriedly ; new offset needs to account for existing offset, not start at 0. Currently upload of single chunk works but not upload of multiple chunks.

[Acconut]

You are right, the current calculation of the new offset is incorrect. I adjust the calculation locally but that does not enable uploading of multiple chunks, unfortunately.

FYI, the GCSStore is currently using simple uploads (see https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload) but we would have to use multipart or resumable uploads to fully support the tus protocol. I wasn't aware of this limitation, sorry for that.

In the meantime, I can recommend you to checkout tusd (https://github.com/tus/tusd) which has a fully working and tested storage backend using GCS.

[Murderlon]

This should be fixed by now :)

[aqib008]

Hi, i am try to upload files using presigned url to GCS, can someone help me out or provide me an sample example. I am using tus js client, it would be very helpful if someone can provide details for implementation

[Acconut]

@aqib008 Please don't spam other discussions with your unrelated question from https://github.com/tus/tus-js-client/issues/616.