How not to have the source files collected in `STATIC_ROOT` ?

[blaze33]

Currently upgrading from django-pipeline 1.3.27 to 1.6.6.

The new default configuration works fine except that I ended with too many files collected in STATIC_ROOT.

Ok, so heading to the documentation, storage section:

If you want to exclude Pipelinable content from your collected static files, you can also use Pipeline’s FileSystemFinder and AppDirectoriesFinder. These finders will also exclude unwanted content like READMEs, tests and examples, which are particularly useful if you’re collecting content from a tool like Bower.

Nice, so I'll try using pipeline.finders.FileSystemFinder and pipeline.finders.AppDirectoriesFinder instead of their django versions but now the compiling / compressing part no longer works because the compilers / compressors are looking for the source files in STATIC_ROOT where there are no longer collected as wanted.

So now I collect the files I need + the source files, then run pipeline and finally remove the source files from STATIC_ROOT but it just feels not right. (plus it's not really fixed, if I have source.scss compiled to source.css compressed to output.css, I'm still ending up with an unwanted source.css in STATIC_ROOT).

Previously we had pipeline.storage.PipelineFinderStorage who used all the finders without the ignore_patterns to find and use all the source files no matter if they were collected in STATIC_ROOT in the end. This Storage was removed when pipeline.collector.default_collector was introduced (I'm still confused about it's purpose) and replaced by django's stock staticfiles_storage who cannot find the source files if they're not copied in STATIC_ROOT.

Related issues: https://github.com/jazzband/django-pipeline/issues/504 https://github.com/jazzband/django-pipeline/issues/503 but they have not attracted any response.

I'll would be glad if someone could point me in the right direction. Am I wrongly using django-pipeline ? Because if not, at that point I may as well directly fix django-pipeline than hack around it but I would need some guidelines from a maintainer before starting to do so.

[blaze33]

I fixed my issue of not being able to control the content of STATIC_ROOT with the following code.

It's essentially a Finder mixin to collect in STATIC_ROOT the extra files needed by the compiling / compressing step and an overriden PipelineStorage that cleans those unwanted files from STATIC_ROOT afterwards.

If anyone has a better solution...

# -*- coding: utf-8 -*-

from __future__ import unicode_literals

import os

from django.contrib.staticfiles import utils

import sh
from pipeline.storage import PipelineStorage
from pipeline.conf import settings as pipeline_settings


def prefix_path(path, storage):
    """
    Prefix the relative path if the source storage contains it
    """
    if getattr(storage, 'prefix', None):
        prefixed_path = os.path.join(storage.prefix, path)
    else:
        prefixed_path = path
    return prefixed_path


class PipelineManifest(object):
    """
    Keep track of django-pipeline packages inputs / outputs files
    """
    _sources = None

    @property
    def sources(self):
        if self._sources is None:
            self._sources = [
                source_filename
                for value in pipeline_settings.STYLESHEETS.values() + pipeline_settings.JAVASCRIPT.values()
                for source_filename in value["source_filenames"]
            ]
        return self._sources

    _outputs = None

    @property
    def outputs(self):
        if self._outputs is None:
            self._outputs = [
                value["output_filename"]
                for value in pipeline_settings.STYLESHEETS.values() + pipeline_settings.JAVASCRIPT.values()
            ]
        return self._outputs

    source_dependencies = [".less"]

    def is_dependency(self, path):
        return os.path.splitext(path)[1] in self.source_dependencies


pipeline_manifest = PipelineManifest()  # pylint: disable=invalid-name


class CleanedPipelineStorage(PipelineStorage):
    """
    Extends PipelineStorage to clean STATIC_ROOT from the extra files collected
    by PipelineFinderMixin
    """
    def post_process(self, paths, dry_run=False, **options):
        if dry_run:
            return

        hashed_names = []
        for name, hashed_name, processed in super(CleanedPipelineStorage, self).post_process(paths.copy(), dry_run, **options):
            hashed_names.append(hashed_name)
            yield name, hashed_name, processed

        self.clean(paths, hashed_names)

    def clean(self, collected_files, processed_files):
        # delete pipeline source files, we only want to collect the output of pipeline
        for path in pipeline_manifest.sources:
            if path not in pipeline_manifest.outputs:  # do not delete inputs already replaced by their output
                print "Deleting pipeline source {}".format(path)
                self.delete(path)

        for path in utils.get_files(self, []):
            # delete intermediary files created by post_process like:
            # `source.scss` compiled to `source.css` compressed to `output.css`
            # source.scss is deleted by above code but not source.css
            if path not in collected_files and path not in processed_files:
                print "Deleting pipeline intermediary file {}".format(path)
                self.delete(path)
            # delete additional pipeline dependencies (like .less files)
            elif pipeline_manifest.is_dependency(path):
                print "Deleting pipeline source dependency {}".format(path)
                self.delete(path)

        # delete empty folders who only contained deleted files
        sh.find(".", "-type", "d", "-empty", "-delete")


class PipelineFinderMixin(object):
    """
    A mixin to extend your STATICFILES_FINDERS to collect files
    needed by django-pipeline at build time
    """
    def list(self, ignore_patterns):
        # collect pipeline sources actually used
        for path, storage in super(PipelineFinderMixin, self).list([]):
            if (prefix_path(path, storage) in pipeline_manifest.sources or
                    pipeline_manifest.is_dependency(path)):
                yield path, storage
        for path, storage in super(PipelineFinderMixin, self).list(ignore_patterns):
            yield path, storage

[blaze33]

Actually I also needed the following Finder to serve the static file during development as pipeline.PipelineFilter (I'm filtering some files with first two filters):

from django.contrib.staticfiles.finders import BaseStorageFinder
from django.contrib.staticfiles.storage import staticfiles_storage

class PipelineFinder(BaseStorageFinder):
    storage = staticfiles_storage

    def list(self, ignore_patterns):
        return []

[meshy]

Also related to https://github.com/jazzband/django-pipeline/pull/233 I think

[kmeht]

EDIT: Apologies, I commented a bit too soon. I believe the lack of updating on refresh is an issue on our end somewhere.

Original Comment: We just upgraded from 1.3.24 to 1.6.8 and are also experiencing the same issue. The files are collected to STATIC_ROOT on the first request, and modifications to the original files (in the app directories) are no longer caught on page refresh. It now appears we must run collectstatic any time we change our static files for pipeline to pick up the updates.

Guessing at maintainers: Was there a change in how we're supposed to use Pipeline? Do we have to make any configuration change to have the previous behavior (e.g. a page refresh should recompile updated files)?

[meshy]

Also related to https://github.com/jazzband/django-pipeline/issues/566

[czubik8805]

In the older version of pipeline was possible to call collectstatic command with ignore parameter e.g:

./manage.py collectstatic --ignore=*.sass --ignore=*.coffee

But now it throws exception

pipeline.exceptions.CompilerError: ['/usr/local/bin/sass', u'/home/michal/project/static/css/styles_commons.sass', u'/home/michal/project/static/css/styles_commons.css'] exit code 1
Errno::ENOENT: No such file or directory @ rb_sysopen - /home/michal/project/static/css/styles_commons.sass
  Use --trace for backtrace.

[viciu]

@cyberdelia could you please help with this issue or at least provide some hints how you would approach this?

[harmon]

Yeah, I just upgraded from 1.3x and things broke for me in dev. I had to set:

DEBUG = True PIPELINE['PIPELINE_ENABLED'] = False PIPELINE['PIPELINE_COLLECTOR_ENABLED'] = True

The switch to forcing files to be copied to STATIC_ROOT instead of just read from the original source dirs seems like an bad change. Every request requires it all to be recopied. It's pretty slow in dev now. AND it copies in all the .scss and .coffee files, since they need to be there for compilation, but aren't cleaned up afterwards. ACK!

The issue is this change from compiling from the source file path to compiling once it's in STATIC_ROOT: https://github.com/jazzband/django-pipeline/commit/bd6b9d8966a5e00701d3a803c4977f20df7a282d#diff-54b8c27056913aafe149e01ff0aa5e46L35

I'd love to see a bit better documentation on how to setup a development environment.