django-pipeline
django-pipeline copied to clipboard
jazzband
"No matching file found…" when PIPELINE_ENABLED = True and DEBUG = True
I'm not sure if this is a bug or intended behavior but I couldn't find anything that specifically answered this question. I'm trying to serve the compiled staticfiles through runserver for debugging. When DEBUG = True and PIPELINE_ENABLED is its default value, I can run ./manage.py findstatic css/generated.css and it will return the correct path.
However, if I override PIPELINE_ENABLED to be True and then run ./manage.py findstatic css/generated.css it returns:
No matching file found for 'css/generated.css'.
Is this intended? It would be great to be able to test compression inside of runserver.
Hi,
I am finding a potentially related issue:
I have I activated the STATICFILES_STORAGE to pipeline.storage.PipelineStorage and set the STATIC_ROOT where minified and non minified versions of my files live. I also have configured the finders to include: pipeline.finders.PipelineFinder
If I have DEBUG True and PIPELINE_ENABLED True I get a systematic 404. Digging into it it seems that Django expects the PipelineFinder not to return [] as it is configured to do when piping is off.
I made my own finder deriving from the PipelineFinder and just have it skip the boolean check for PIPELINE_ENABLED and it seems that everything works as expected then.
This only happens when using STATIC_ROOT to serve local files (dev mode).
@TylerLubeck
Hey,
Sorry, I meant to do a pull request and did not get around to it. Right now, I monkey patched it.
Here is what I have (working). I had to make a bunch of changes to get it work with a STATIC_URL in S3.
I need to make the pull request but in the mean time it should get anyone who runs into this unstuck.
class FixedPipelineFinder(PipelineFinder): # temporary, I believe the library has a bug.
def find(self, path, all=False):
return BaseStorageFinder.find(self, path, all)
# monkey patching the flawed library
def __decorate_sources(func):
def sources(self):
if not self._sources:
paths = []
for pattern in self.config.get('source_filenames', []):
len_path = len(paths)
for path in glob(pattern):
if path not in paths and find(path):
paths.append(str(path))
# BEGIN CHANGE
if len_path == len(paths) and pattern not in paths: # double check
path = find(pattern)
if path:
paths.append(str(pattern))
# END CHANGE
self._sources = paths
return self._sources
return property(fget=sources)
packager.Package.sources = __decorate_sources(packager.Package.sources)
def __read_bytes(self, path): # copied from a pull request off github...
"""Read file content in binary mode"""
finder_path = finders.find(path)
if finder_path is not None:
file = open(finder_path)
else:
raise Exception("File '%s' not found via "
"static files finders", path)
content = file.read()
file.close()
return content
__construct_asset_path_orign = compressors.Compressor.construct_asset_path
def __construct_asset_path(self, asset_path, css_path, output_filename, variant=None):
if settings.STATIC_ROOT is None: # added dirty handling for None static root.
public_path = self.absolute_path(asset_path, os.path.dirname(css_path).replace('\\', '/'))
if public_path[0] == '/' and public_path.startswith('/' + self.storage.location):
public_path = public_path[len(self.storage.location) + 2:]
return self.storage.url(public_path)
return __construct_asset_path_orign(self, asset_path, css_path, output_filename, variant)
compressors.Compressor.read_bytes = __read_bytes
compressors.Compressor.construct_asset_path = __construct_asset_path
def __compile(self, paths, force=False):
def _compile(input_path):
for compiler in self.compilers:
compiler = compiler(verbose=self.verbose, storage=self.storage)
if compiler.match_file(input_path):
output_path = self.output_path(input_path, compiler.output_extension)
# BEGIN CHANGE
try:
infile = finders.find(input_path) # this is different for the original
except NotImplementedError:
infile = finders.find(input_path)
try:
outfile_ = self.storage.path(input_path)
except NotImplementedError:
outfile_ = finders.find(input_path)
# END CHANGE
outfile = self.output_path(outfile_, compiler.output_extension)
outdated = True or compiler.is_outdated(input_path, output_path)
compiler.compile_file(quote(infile), quote(outfile),
outdated=outdated, force=force)
return output_path
else:
return input_path
try:
import multiprocessing
from concurrent import futures
except ImportError:
return list(map(_compile, paths))
else:
with futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count()) as executor:
return list(executor.map(_compile, paths))
compilers.Compiler.compile = __compile
I was running into the same issue. Looking through the source of PipelineFinder I realized we're probably not really meant to use it as it seems to do a NO-OP:
class PipelineFinder(BaseStorageFinder):
storage = staticfiles_storage
def find(self, path, all=False):
if not settings.PIPELINE_ENABLED:
return super(PipelineFinder, self).find(path, all)
else:
return []
def list(self, ignore_patterns):
return []
If PIPELINE_ENABLED is set to False, it simply returns the results from the BaseStorageFinder which is in line with what @ctbarna described, where it kind of works because the assets are in the folder and get eventually found by the BaseStorageFinder. (When it's set to True, it returns an empty list.)
Extending the PipelineFinder and overriding this condition doesn't feel like the way to go. Looking further down in the same file, I found out the ManifestFinder, which is probably what we should be using, rather than the PipelineFinder.
The ManifestFinder does the following:
class ManifestFinder(BaseFinder):
def find(self, path, all=False):
"""
Looks for files in PIPELINE.STYLESHEETS and PIPELINE.JAVASCRIPT
"""
matches = []
for elem in chain(settings.STYLESHEETS.values(), settings.JAVASCRIPT.values()):
if elem['output_filename'] == path:
match = safe_join(settings.PIPELINE_ROOT, path)
if not all:
return match
matches.append(match)
return matches
def list(self, *args):
return []
As we can see, it pulls the values from the STYLESHEETS and JAVASCRIPT dictionaries as defined in the settings.py files. This seems to me like the finder we should be using. The PipelineFinder was probably an abstract class, never really meant to be used.
I have figured out another (probably ugly) hack, but it works in only 4 lines of code:
from django.contrib.staticfiles.finders import BaseStorageFinder
from django.contrib.staticfiles.storage import staticfiles_storage
class FixedPipelineFinder(BaseStorageFinder):
storage = staticfiles_storage
Then use this FixedPipelineFinder in your settings (I put the finder into a package hacks.pipeline):
STATICFILES_FINDERS = (
...
'hacks.pipeline.FixedPipelineFinder',
)
The FixedPipelineFinder is the same as the pipeline.finder.PipelineFinder but with the checks removed from find and list. Now I can test the pipeline in DEBUG=True using PIPELINE_ENABLED=True in my settings.
I don't know the details of how pipeline works, so this might as well break your production build, ruin your marriage or melt the earth.
There was a change in the Compiler that now (as of 1.4.0) uses the storage to look up the input file before processing, instead of the finder. This means that the input will need to come from the STATIC_ROOT end result of running collectstatic, and not from the STATICFILES_DIRS, where the source files come from: https://github.com/jazzband/django-pipeline/commit/bd6b9d8966a5e00701d3a803c4977f20df7a282d#diff-54b8c27056913aafe149e01ff0aa5e46L35
After this change, it basically requires that the files first are copied into STATIC_ROOT, whereas before they could just be sourced from the original directories. The only way I can get DEBUG=True to work without first running collectstatic is like this:
DEBUG = True
PIPELINE = {
''PIPELINE_ENABLED": False,
"PIPELINE_COLLECTOR_ENABLED": True
}
It seems like this change in 1.4x made it required in development to copy all files into the STATIC_ROOT on each request, which adds a huge penalty. I wish it would just load out of the source folders like in 1.3.
Another thing, is that if PIPELINE_ENABLED = True, it expects STATIC_ROOT to have files. When it's turned off, but you have PIPELINE_COLLECTOR_ENABLED = True, it will generate the STATIC_ROOT for you, so that's why it works. But setting PIPELINE_ENABLED = True skips the check for PIPELINE_COLLECTOR_ENABLED, so it can't generate the STATIC_ROOT on the fly.
See:
https://github.com/jazzband/django-pipeline/blob/master/pipeline/templatetags/pipeline.py#L67-L72
And:
https://github.com/jazzband/django-pipeline/blob/master/pipeline/templatetags/pipeline.py#L97-L98
I'm having the same issue outlined in the original post. When I'm in DEBUG=True I sometimes want to have pipeline serve compiled/concatenated assets. If I override the PIPELINE_ENABLED=True the compiled assets are properly referenced the response, but server returns a 404 error for those assets.
If DEBUG=False and I have previously run collectstatic I have no problems (compiled versions are generated and served). If I don't override the PIPELINE_ENABLED, I also have no problems -- but individual source files are served rather than the compiled version.
django-pipeline=1.6.12
