django-storages icon indicating copy to clipboard operation
django-storages copied to clipboard

`collectstatic` is extremely slow with S3 Manifest

Open catgirlinspace opened this issue 1 year ago • 5 comments

hiya! I'm deploying my app on DigitalOcean App Platform. As part of the build step there, I run python manage.py collectstatic --noinput. Before switching to storing my static assets in a S3-compatible bucket this was instant, but now takes over 10 minutes when using the S3 manifest. The S3-compatible service I am using is Backblaze B2, although I wouldn't expect this to matter.

Configuration for my static files

storages.py

class StaticStorage(S3ManifestStaticStorage):
    bucket_name = 'splashcat-static'
    custom_domain = 'static.splashcat.ink'

settings.py

if not DEBUG:
    STORAGES = global_settings.STORAGES | {
        "default": {"BACKEND": "storages.backends.s3boto3.S3Boto3Storage"},
        "staticfiles": {"BACKEND": "splashcat.storages.StaticStorage"}
    }

catgirlinspace avatar Jun 02 '23 21:06 catgirlinspace

@catgirlinspace I am experiencing this too, did you find a solution?

DuncanTormey avatar Oct 26 '23 17:10 DuncanTormey

I ended up doing this.

python -m cosmos.manage collectstatic --no-input & # background the static collection
gunicorn cosmos.cosmos_site.wsgi --config cosmos/gunicorn_conf.py

DuncanTormey avatar Oct 26 '23 22:10 DuncanTormey

@DuncanTormey hi, sorry for the late reply. my workaround for this was to just use collectstatic and serve it without S3 and then stick a CDN in front.

catgirlinspace avatar Nov 09 '23 21:11 catgirlinspace

I'm also experiencing a 10 minute duration when trying to upload ~2,700 files

robd003 avatar Mar 08 '24 03:03 robd003

+1

davidmir avatar Apr 10 '24 13:04 davidmir