tator icon indicating copy to clipboard operation
tator copied to clipboard

Incompatibility with Min.IO when transaction size is 64Kb

Open bctcvai opened this issue 11 months ago • 1 comments

The multibucket API wrapper in store.py appears to have an API in compatibility with min.io servers.

Steps to reproduce:

1.) Configure a min.io project specific bucket. 2.) Attempt to upload a file in 64Kb chunks 3.) Observe the upload fails.

In the REST logs:

INFO 2024-03-23 12:14:05,716 store 9802 140398682389568 "Multipart failed: An error occurred (EntityTooSmall) when calling the CompleteMultipartUpload operation: Your proposed upload is smaller than the minimum allowed object size."
INFO 2024-03-23 12:14:05,716 _base_views 9802 140398682389568 "Handling Exception!"
INFO 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "<class 'Exception'>"
ERROR 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "Exception in request: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/tator_online/main/rest/_base_views.py", line 63, in post
    response_data = self._post(params)
  File "/tator_online/main/rest/upload_completion.py", line 35, in _post
    raise Exception(f"Upload completion for {key} failed!")
Exception: Upload completion for XX/YYY/ZZZZZZZZZ/720_ftuxqKvUKv.mp4 failed!

Client-side instrumentation, it can be observed there are 45 parts making up the upload, each with a valid ETag assigned.

bctcvai avatar Mar 23 '24 12:03 bctcvai

We should add checks for valid chunk size, as to fail more gracefully.

Root cause of issue was default timeout of 30 seconds was added/documented but didn't change/break API so it was a silent but deadly change to cellular use cases.

bctcvai avatar Mar 23 '24 19:03 bctcvai