compilers do not use django storage to save files

[miki725]

For example when using S3 to serve static files with combination of less compiler, none of the less converted files are uploaded to S3 since less is a subprocess compile which simply calls lessc command which stores the file locally. Am I doing something wrong with using pipeline in combination with S3? I followed suggestions at https://django-pipeline.readthedocs.io/en/latest/storages.html#using-with-other-storages

here is my storage backend:

from django.contrib.staticfiles.storage import ManifestFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
    pass

from quick glance seems issue could be related to https://github.com/jazzband/django-pipeline/blob/master/pipeline/compilers/init.py#L40 since that simply calls the compiler but later on does not use the storage to save that compiled result via django storage.

any ideas?

[miki725]

seems this is actually related when compiler is used with compressor. so lessc creates file locally but then when attempting to compress that file, compressor attempts to read it via storage which does not have it yet.

here is exact traceback btw:

Traceback (most recent call last):
  File "./manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
    collected = self.collect()
  File "/usr/local/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect
    for original_path, processed_path, processed in processor:
  File "/usr/local/lib/python3.5/site-packages/pipeline/storage.py", line 26, in post_process
    packager.pack_stylesheets(package)
  File "/usr/local/lib/python3.5/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/pipeline/packager.py", line 107, in pack
    content = compress(paths, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 75, in compress_css
    css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 139, in concatenate_and_rewrite
    content = self.read_text(path)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 223, in read_text
    content = self.read_bytes(path)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 217, in read_bytes
    file = staticfiles_storage.open(path)
  File "/usr/local/lib/python3.5/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.5/site-packages/storages/backends/s3boto.py", line 393, in _open
    raise IOError('File does not exist: %s' % name)
OSError: File does not exist: somepath/foo.css

[miki725]

so the solution would probably be somehow would have to save all the paths before compression in https://github.com/jazzband/django-pipeline/blob/master/pipeline/packager.py#L105

[miki725]

just sent PR with a fix which seems to work - https://github.com/jazzband/django-pipeline/pull/609

[pythdasch]

Hi I think I have the same issue as you @miki725 . How did you make it work ?

[miki725]

@pythdasch just using the fork at the moment

[pythdasch]

thanks :)

[pythdasch]

@miki725 Not working for me. I guess it's not a storage problem, but more a compiler problem. so annoying

[SverkerSbrg]

+1

[sharov]

I've sent PR #663, which work with original Django storages & storages from package 'django-storages'