tator
tator copied to clipboard
Incompatibility with Min.IO when transaction size is 64Kb
The multibucket API wrapper in store.py appears to have an API in compatibility with min.io servers.
Steps to reproduce:
1.) Configure a min.io project specific bucket. 2.) Attempt to upload a file in 64Kb chunks 3.) Observe the upload fails.
In the REST logs:
INFO 2024-03-23 12:14:05,716 store 9802 140398682389568 "Multipart failed: An error occurred (EntityTooSmall) when calling the CompleteMultipartUpload operation: Your proposed upload is smaller than the minimum allowed object size."
INFO 2024-03-23 12:14:05,716 _base_views 9802 140398682389568 "Handling Exception!"
INFO 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "<class 'Exception'>"
ERROR 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "Exception in request: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/tator_online/main/rest/_base_views.py", line 63, in post
response_data = self._post(params)
File "/tator_online/main/rest/upload_completion.py", line 35, in _post
raise Exception(f"Upload completion for {key} failed!")
Exception: Upload completion for XX/YYY/ZZZZZZZZZ/720_ftuxqKvUKv.mp4 failed!
Client-side instrumentation, it can be observed there are 45 parts making up the upload, each with a valid ETag assigned.
We should add checks for valid chunk size, as to fail more gracefully.
Root cause of issue was default timeout of 30 seconds was added/documented but didn't change/break API so it was a silent but deadly change to cellular use cases.