codalab-competitions icon indicating copy to clipboard operation
codalab-competitions copied to clipboard

"Many existing datasets bundled into one new dataset" option not working when creating a dataset

Open Didayolo opened this issue 4 years ago • 2 comments

The option "Many existing datasets bundled into one new dataset" in "Create dataset"...

codalab-many-existing

...returns the following error:

The CodaLab site is currently unavailable.

Sorry, the page you are looking for is currently unavailable.

Please try again later.

Didayolo avatar Jan 04 '22 19:01 Didayolo

The error message linked to this seems to be this one:

django            | 2022-01-17 14:57:07,609 INFO Calculating sub size for submission: 35361
django            | 2022-01-17 14:57:07,609 apps.web.models [INFO] Calculating sub size for submission: 35361
django            | 2022-01-17 14:57:08,878 apps.web.tasks [INFO] Path is: phase_input_data_file/7852785e-7db3-4a87-9650-097a4c0ce87c/competition/909/3/data/input.zip
django            | 2022-01-17 14:57:08,878 INFO Path is: phase_input_data_file/7852785e-7db3-4a87-9650-097a4c0ce87c/competition/909/3/data/input.zip
django            | 2022-01-17 14:57:08,891 apps.web.tasks [INFO] Path is: starting_kit/6c432043-aa53-450c-862c-856f43830bbd/competition/909/4/data/starting_kit.zip
django            | 2022-01-17 14:57:08,891 INFO Path is: starting_kit/6c432043-aa53-450c-862c-856f43830bbd/competition/909/4/data/starting_kit.zip
django            | 2022-01-17 14:57:08,894 ERROR Internal Server Error: /my/datasets/create
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 41, in inner
django            |     response = get_response(request)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 187, in _get_response
django            |     response = self.process_exception_by_middleware(e, request)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 185, in _get_response
django            |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 68, in view
django            |     return self.dispatch(request, *args, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/mixins.py", line 56, in dispatch
django            |     return super(LoginRequiredMixin, self).dispatch(request, *args, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 88, in dispatch
django            |     return handler(request, *args, **kwargs)
django            |   File "/app/codalab/apps/web/views.py", line 1531, in post
django            |     return self.form_valid(form)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/edit.py", line 162, in form_valid
django            |     self.object = form.save()
django            |   File "/app/codalab/apps/web/forms.py", line 256, in save
django            |     instance.write_multidataset_metadata(datasets=self.cleaned_data.get("sub_data_files"))
django            |   File "/app/codalab/apps/web/models.py", line 2535, in write_multidataset_metadata
django            |     self.data_file.save("metadata", ContentFile("\n".join(lines)))
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 94, in save
django            |     self.name = self.storage.save(name, content, max_length=self.field.max_length)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
django            |     return self._save(name, content)
django            |   File "/usr/local/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 547, in _save
django            |     obj.upload_fileobj(content, ExtraArgs=params)
django            |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj
django            |     return self.meta.client.upload_fileobj(
django            |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
django            |     return future.result()
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
django            |     return self._coordinator.result()
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
django            |     raise self._exception
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
django            |     return self._execute_main(kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
django            |     return_value = self._main(**kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
django            |     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 337, in _api_call
django            |     return self._make_api_call(operation_name, kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 632, in _make_api_call
django            |     handler, event_response = self.meta.events.emit_until_response(
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 360, in emit_until_response
django            |     return self._emitter.emit_until_response(aliased_event_name, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 243, in emit_until_response
django            |     responses = self._emit(event_name, kwargs, stop_on_response=True)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
django            |     response = handler(**kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1769, in conditionally_calculate_md5
django            |     md5_digest = calculate_md5(body, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1746, in calculate_md5
django            |     binary_md5 = _calculate_md5_from_file(body)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1759, in _calculate_md5_from_file
django            |     md5.update(chunk)
django            | TypeError: Unicode-objects must be encoded before hashing
django            | 2022-01-17 14:57:08,894 django.request [ERROR] Internal Server Error: /my/datasets/create
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 41, in inner
django            |     response = get_response(request)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 187, in _get_response
django            |     response = self.process_exception_by_middleware(e, request)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 185, in _get_response
django            |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 68, in view
django            |     return self.dispatch(request, *args, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/mixins.py", line 56, in dispatch
django            |     return super(LoginRequiredMixin, self).dispatch(request, *args, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 88, in dispatch
django            |     return handler(request, *args, **kwargs)
django            |   File "/app/codalab/apps/web/views.py", line 1531, in post
django            |     return self.form_valid(form)
django            |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/edit.py", line 162, in form_valid
django            |     self.object = form.save()
django            |   File "/app/codalab/apps/web/forms.py", line 256, in save
django            |     instance.write_multidataset_metadata(datasets=self.cleaned_data.get("sub_data_files"))
django            |   File "/app/codalab/apps/web/models.py", line 2535, in write_multidataset_metadata
django            |     self.data_file.save("metadata", ContentFile("\n".join(lines)))
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 94, in save
django            |     self.name = self.storage.save(name, content, max_length=self.field.max_length)
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
django            |     return self._save(name, content)
django            |   File "/usr/local/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 547, in _save
django            |     obj.upload_fileobj(content, ExtraArgs=params)
django            |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj
django            |     return self.meta.client.upload_fileobj(
django            |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
django            |     return future.result()
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
django            |     return self._coordinator.result()
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
django            |     raise self._exception
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
django            |     return self._execute_main(kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
django            |     return_value = self._main(**kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
django            |     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 337, in _api_call
django            |     return self._make_api_call(operation_name, kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 632, in _make_api_call
django            |     handler, event_response = self.meta.events.emit_until_response(
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 360, in emit_until_response
django            |     return self._emitter.emit_until_response(aliased_event_name, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 243, in emit_until_response
django            |     responses = self._emit(event_name, kwargs, stop_on_response=True)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
django            |     response = handler(**kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1769, in conditionally_calculate_md5
django            |     md5_digest = calculate_md5(body, **kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1746, in calculate_md5
django            |     binary_md5 = _calculate_md5_from_file(body)
django            |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 1759, in _calculate_md5_from_file
django            |     md5.update(chunk)
django            | TypeError: Unicode-objects must be encoded before hashing

Didayolo avatar Jan 17 '22 14:01 Didayolo

@dtuantran this error may be linked to boto too.

Didayolo avatar Jan 17 '22 14:01 Didayolo