Need an option to exclude files from precaching

[jpmedley]

trafficstars

If I'm building a new app, I can structure my site such that I can easily separate files to be cached from those that shouldn't. If I'm adding precache to an existing site, the separation may not be that simple. I might have paths in build scripts for example that aren't easily changed.

I can imagine two possible implementations for this. It could be a member of the options object or it could be a list in a file called .precache-ignore (or something like that).

[jpmedley]

Here's an example. Let's take the app todomvc. To build this site, I need to install some node modules. A cache statement such as the following will pick up my node modules.

staticFileGlobs: [rootDir + '/**/*.{js,html,css,png,jpg,gif}']

To get around that I need

staticFileGlobs: [rootDir + '/media/*.{js,html,css,png,jpg,gif}',
                  rootDir + '/site-assets/*.{js,html,css,png,jpg,gif}',
                  // etc. ]

What's worse is there's a folder called examples/ that has a node_modules/ in each of 73 different folders.

(I grant you that this last is a bit absurd. It's material that probably wouldn't be precached in any scenario. But what if...)

[jeffposnick]

@jpmedley in the TodoMVC case, as I think is common in many projects, there's a build process that outputs the "production" version of the site into a directory. TodoMVC puts everything into dist/ once you build it with gulp. So if you used dist/ as your rootDir when configuring sw-precache, it would only pick up the local assets that are actually used in the production version of your site.

The patterns that you can pass in to sw-precache are flexible, and correspond directly to what's supported by underlying glob module. If you were in a situation in which you needed to precache a subset of directories' contents while excluding others, there's a concise way of specifying those alternatives:

staticFileGlobs: [rootDir + '/{dir1,dir2,dir3}/**/*.{js,html,css,png,jpg,gif}']

This is a whitelist rather than a blacklist, but I think it handles common use cases.

[jpmedley]

@jeffposnick I'll stick a line or two in the readme then.

[mukilane]

I just used the glob negation to exclude files or directories staticFileGlobs: [rootDir + '!dir1{,/**}'] This excludes the dir1 directory and its contents.

[humphd]

I'm trying to do something that works elsewhere I use glob, but doesn't seem to here--perhaps I'm misunderstanding. Given a dir dist/, I want to include everything it has except one of its sub-dirs, for example dist/a. I would assume this would work:

staticFileGlobs: [
  'dist/**',
  '!dist/{a,a/**}'
]

No matter how I try to rewrite the ignore pattern, I can't seem to get it. I tried writing a test: given your test/data/one dir, I would expect both of these to only cache test/data/one/a.txt:

  it('should produce same output via ignore glob as explicitly passing to staticFileGlobs', function(done) {
    var config = {
      logger: NOOP,
      staticFileGlobs: [
        'test/data/one/*.txt',
        '!test/data/one/c.txt'
      ]
    };

    var configPrime = {
      logger: NOOP,
      staticFileGlobs: [
        'test/data/one/a.txt'
      ]
    };

    generate(config, function(error, responseString) {
      assert.ifError(error);
      generate(configPrime, function(error, responseStringPrime) {
        assert.ifError(error);
        assert.strictEqual(responseString, responseStringPrime);
        done();
      });
    });
  });

However, it fails like so:

  1 failing

  1) sw-precache core functionality should produce same output via ignore glob as explicitly passing to staticFileGlobs:

      Uncaught AssertionError: '/**\n * Copyright 2016 Google Inc. All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the "License"); === '/**\n * Copyright 2016 Google Inc. All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the "License");
      + expected - actual

       /* eslint-env worker, serviceworker */
       /* eslint-disable indent, no-unused-vars, no-multiple-empty-lines, max-nested-callbacks, space-before-function-paren, quotes, comma-spacing */
       'use strict';

      -var precacheConfig = [["test/data/one/a.txt","60b725f10c9c85c70d97880dfe8191b3"],["test/data/one/c.txt","2cd6ee2c70b0bde53fbe6cac3c8b8bb1"]];
      +var precacheConfig = [["test/data/one/a.txt","60b725f10c9c85c70d97880dfe8191b3"]];
       var cacheName = 'sw-precache-v3--' + (self.registration ? self.registration.scope : '');


       var ignoreUrlParametersMatching = [/^utm_/];

      at test/test.js:153:16
      at lib/sw-precache.js:312:9
      at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)



{ [Error: 1 test failed.]
  message: '1 test failed.',
  showStack: false,
  showProperties: true,
  plugin: 'gulp-mocha',
  __safety: { toString: [Function: bound ] } }
[23:15:50] npm ERR! Test failed.  See above for more details.

Is this a bug, or am I just assuming things that aren't true? You mention whitelisting above, and I'd love to avoid this, since my build dir can change a lot; with a blacklist added, I'm covered. Thanks for clarifying.

[jeffposnick]

Can you try it with: staticFileGlobs: [rootDir + '/!(a)/**/*']

I can't claim to fully understand the glob syntax, but that seems to work in some of my local experimentations.

It might also have something to do with the version of glob (and its dependencies) that we're using in sw-precache.

[Akta3d]

Yes, it will be really useful to have possibility to write

 staticFileGlobs: [
        'img/**/*',
        '!img/no-offline/**/*',
         ...
      ]

[gravi2]

I think glob has stopped supporting the comments and negation syntax. More info at the link below: https://github.com/isaacs/node-glob#comments-and-negation

[humphd]

That's interesting, I hadn't seen they'd moved those into ignore. Though, they've only moved where you define it, not the concept of eliminating paths based on a pattern. I still think it would be useful here, however it gets done.

[OP-Klaus]

It's important to be able to blacklist instead of whitelist for future-proofing complex applications. It would be great to see an option like staticFileGlobsIgnores

[mbj36]

Hi all I have a root directory app , inside app there is bower_components directory. My staticFileGlobs looks like this staticFileGlobs: [ rootDir + '/**/*.{js,html,css,scss,png,jpg,gif,svg,eot,ttf,woff,json}', rootDir + '!/bower_components', rootDir + '!/bower_components/**' ]

I don't want to cache bower_components directory but this is not working. please help

[ngie-mp]

Hi you all,

I'm trying to exclude some files from my stack cache list.. when I generate the sw-precache the ! prefix is not supported so, I still having the undesired files on cache instead.. any idea how I can exclude some specific files from sw-precache someone? Pleaaase!!

module.exports = {
  staticFileGlobs: [
   'app/**/*.css',
    'app/**/*.html',
    'app/**/*.js',
    'app/**/*.js',
    'app/**/*.json',
    'app/**/*.{jpg,svg,png}',
    'app/fonts/*.{woff,woff2,ttf}',
    '!app/vendor/angular-mocks.js',
    '!app/js/app_test.js'
  ],
  root: 'app'
};

[marie-dk]

I finally found out how to exclude dirs and files from staticFileGlobs. My directory is like this:

src
  build
    gfx
    css
    fonts

My rootDir is src/build

I want to cache everything but the fonts directory and two images in gfx. First I made the mistake of thinking that I should exclude stuff in separate globs, as I am used to do in gulp tasks. But it can alle be done in one line. The following will include everything but exclude the fonts dir, large-img-01 and large-img-02.

staticFileGlobs: [
  rootDir + '/!(fonts)/**/!(large-img-01|large-img-02)*.{js,html,css,png,jpg,gif,svg,eot,ttf,woff}',
],

Hope this helps someone who (as I did) passes by this post in search for an answer...

[Osirisxxl]

#349 adds a working solution to exclude files from precaching. It won't be merged and hasn't received @jeffposnick approval as this repo has gone into maintenance. Anyway, should you need this feature, feel free to check our solution out.

[vivekiyer114]

Here's a quick fix. // navigateFallback: publicUrl + '/index.html',

I just commented navigateFallback function. Any demerits for this please let me know.

[zoxon]

Yet another quick fix

const glob = require("glob");
const path = require("path");

const dist = "dest";

const patterns = [
  `${dist}/**/*.html`,
  `${dist}/**/*.js`,
  `${dist}/**/*.css`,
  `${dist}/assets/**`
];

const paths = [];
patterns.forEach(pattern =>
  paths.push(...glob.sync(pattern.replace(path.sep, "/")))
);

const config = {
  navigateFallback: "/index.html",
  stripPrefix: `${dist}/`,
  staticFileGlobs: paths.filter(path => !/assets\/images\/content/.test(path))
};