Clear fails with S3Boto3Storage

[WhyNotHugo]

Trying to clear a S3Boto3Storage fails (at least when using it for static files):

$ django-admin collectstatic  -c
[    INFO 17:08:26] raven.contrib.django.client.DjangoClient:215  Raven is not configured (logging is disabled). Please see the documentation for more information.

You have requested to collect static files at the destination
location as specified in your settings.

This will DELETE ALL FILES in this location!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
  File "/usr/local/bin/django-admin", line 11, in <module>
    sys.exit(execute_from_command_line())
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
    collected = self.collect()
  File "/usr/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 106, in collect
    self.clear_dir('')
  File "/usr/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 233, in clear_dir
    if not self.storage.exists(path):
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 478, in exists
    self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 312, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 579, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 634, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python3.6/site-packages/botocore/validate.py", line 291, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter Key, value: 0, valid range: 1-inf

[aguynamedben]

+1. I experienced this error and it sent me on a 1-hour red herring because I suspected the cause was more intricate logic in collectstatic. Then I realized it was just the --clear option.

FWIW, I was using a hybrid backend like this one

# storage.py
class StaticStorage(ManifestFilesMixin, S3Boto3Storage):
    pass

# settings.py
STATICFILES_STORAGE = 'storage.StaticStorage'
``

[WhyNotHugo]

Just to update this: I didn't find a simple way around this.

I ended up using whitenoise and cloudfront for static content.

[aguynamedben]

@WhyNotHugo Thanks for the tip, whitenoise + cloudfront and just skipping S3 looks like a good option, I'll look at that.

[petrboros]

Is there any development on this issue? We are still experiencing the problem. Thank you very much.

[serjant]

The problem here too, so I am currently using manual aws command to clear the assets on the CDN:

aws s3 rm s3://<bucket name> --recursive

[jcarizza]

I'm using django-storages==1.9 and Python 3.6 with Boto3.

I investigated this issue and for unknown reasons, a file with the name '' (empty) gets until this function .exists().

When the function try to check the object with Key='' in the bucket it raises an error that is not captured by the except ClientError, this exception is botocore.exceptions.ParamValidationError.

Why an empty name gets so far? I think it should be validated before calling.exists()

https://github.com/jschneier/django-storages/blob/0a93eef19c96b2f7705d23251ecb1ca918fc6b40/storages/backends/s3boto3.py#L556

[rotherfuchs]

Did anyone find out yet where that empty name comes from ?

[aidenbell]

This looks like it is hard-coded in the management command:

  File "/usr/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 106, in collect
    self.clear_dir('')
  File "/usr/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 233, in clear_dir
    if not self.storage.exists(path):

I'm not sure what it should be to work with S3Boto3Storage. I suppose it'd be possible to extend .exists() in a storage engine and skip this ... i'm not actually sure what collectstatic is trying to do with the empty string passed to self.storage.exists('') though ... weird.

Django 3.1.3 django-storages 1.11.1

FYI

[SandervanL]

I also ran into this problem know. Has any fix been found?

django==3.2.13
django-storages==1.12.3
boto3==1.21.8
botocore==1.24.8

[martingaldeca]

I also ran into this problem know. Has any fix been found?

django==3.2.13
django-storages==1.12.3
boto3==1.21.8
botocore==1.24.8

check if you are calling the collectstatic command, if you are, just make sure that you are not calling it with -c or --clear option

[timkofu]

In my S3ManifestStaticStorage subclass, adding a location fixed the issue, and I'm able to continue using --clear.

[nikolas]

Adding this line fixed the issue for me:

AWS_LOCATION = 'media/'