[workbox-webpack-plugin] proposal to add support Webpack assets in `templatedUrls` option

[raejin]

Library Affected: workbox-webpack-plugin

Browser & Platform: N/A

Issue or Feature Request Description: This proposal is derived from a conversation dated back in February. Currently, the templatedUrls option looks like the following:

new InjectManifest({
  swSrc: 'sw.js',
  templatedUrls: {
    '/shell': [
      'path/to/file1/*.js',
      'path/to/file2/*.js',
      'path/to/file3/*.js',
    ],
  },
})

This works well if the shell is based on files that already exist on the disk, which is exposed and captured by the specified glob syntax. However, if a shell is based on some files that are part of the Webpack compilation, we would be out of luck. Since the workbox-webpack-plugin is hooked into the emit Webpack lifecycle, by the time the plugin evaluates the templatedUrls option, the compilation assets won't be available on the disk yet. (This behavior is documented in the Webpack documentation here)

What I'd like to propose, is an API change to the templatedUrls option, so it does support looking into the Webpack compilation. Suggestion is welcome to improve the usability of the API, as I can foreseen confusion introduced with the change.

new InjectManifest({
  swSrc: 'sw.js',
  templatedUrls: {
    '/shell': {
      globPatterns: [
        'path/to/file1/*.js',
        'path/to/file2/*.js',
        'path/to/file3/*.js',
      ],
      chunks: [
        'chunk1',
        'chunk2',
        'chunk3',
      ],
      include: [
        /common.css$/,
      ],
    },
  },
})

The objective of the new support is to ensure that any Webpack compiled asset should be able to be taken into account as to determine the fingerprint of the shell file.

[jeffposnick]

I'm supportive of this change. As you mention, coming up with the right interface is a bit of a challenge. (This same challenge applies to other options that are currently limited to working with the filesystem, but which might make sense when extended to work with webpack assets, like manifestTransforms.)

Your proposal where the webpack-specific behavior would be triggered by passing in an Object rather than an Array as the configuration for a given templatedUrl entry seems clean enough to me. I'm not sure whether that Object would need to support a globPatterns entry, since it seems like we should try to keep the glob-based and webpack-based configuration separate in this case, and you wouldn't need to mix them for a given URL.

I'm assuming that in this model, chunks and include would be interpreted the same way that they are for the top-level webpack plugin config? Do you think there should be any defaults here, or just require everything to be explicitly configured?

[raejin]

@jeffposnick

Sorry for the delayed followup.

The argument for supporting globPatterns in the configuration object is for flexibility. I cannot confidently make the assumption that all of the needed assets only live in a Webpack compilation. Maybe it can be a global font that is not processed by Webapck etc.

I'm assuming that in this model, chunks and include would be interpreted the same way that they are for the top-level webpack plugin config?

Could you elaborate on this? What specific config did you have in mind?

I'm not sure if it makes sense to provide a default. If I were to configure this, I'd expect the default to not include anything in the Webpack compilation.

[Jamiewarb]

My issue from #1487 seems to be an off-shoot of this hopefully. It's something I'm running in to in multiple projects now, so would be ideal to get this looked at. Thanks @jeffposnick !

[zevdg]

I am also affected by this. Specifically, I ran into this bug https://github.com/v8/v8.dev/issues/4 in my own PWA, and part of the fix outlined there involves adding templateUrl config like this https://github.com/v8/v8.dev/pull/7/files

Since my index.html is generated from a template using the standard html webpack plugin, I cannot apply that part of the solution until templateUrl supports assets. Well, I could, but afaict, the content hash would be based on the html template and not the generated index.html which would be very bad.

I'm able to solve the issue by hand editing the precache-manifest file after my project is built like so.

...
{
  revision: "2f0cec2af109b3d99a141e7685f1596e",
  url: "/index.html"
},
// manually add this section, copy the revision hash from above
{
  revision: "2f0cec2af109b3d99a141e7685f1596e",
  url: "/"
}

I can make a post build step that automates this process, but if there's a better option, I'd love to hear it.

edit: my hacky post build step https://gist.github.com/zevdg/552530c973e01c9ff06ed14fb71465a8

[jeffposnick]

FWIW, we still don't have a "native" way of doing templatedURLs in workbox-webpack-plugin, but there's a bit of a cleaner approach possible in v5, using a ManifestTransform function:

https://github.com/GoogleChrome/workbox/issues/2398#issuecomment-597080778

[dancandertons]

@jeffposnick apologies if I've missed some progress on this somewhere (I've done a fair bit of diving around various threads on this subject), but the above workaround still wouldn't address the issue of not being able to reference files which are yet to be compiled, is that correct?

I'm using the workaround in #2398 to add the app shell to my precache manifest, for which I need a revision based on the handlebars template it's based off of, as well as my compiled JS and CSS bundles (complete with hashes). Obviously, at the point in the webpack lifecycle GenerateSW is run, these JS and CSS files don't exist, so the app shell's revision ends up being based solely on a template which rarely changes. This is causing quite a lot of issues - before I start trying to get hacky, I just wanted to check whether you're aware of any workaround which would allow manifestTransforms to see the compiled files?

[jeffposnick]

A manifestTransforms function, when used in workbox-webpack-plugin, are passed a webpack Compilation object as the second parameter. This answer depends somewhat on exactly how your JS and CSS is being generated by webpack, but in general, it's fairly likely that the Compilation "knows" about them, and you'd see them listed in the assets property of the Compilation. workbox-webpack-plugin attempts to run its plugin code fairly late in the webpack compilation process, after (hopefully) all the assets have been generated.

If you're not seeing the JS and CSS in the Compilation's assets, then could you share your webpack configuration? Maybe we can figure out how to reorder things to make them available.

[dancandertons]

Thanks Jeff! That's what I was missing. Written a solution which is probably pretty specific to my application, but I'll share it in case it saves anyone the many hours of confusion I've been through. It's not particularly elegant - I've struggled to find any documentation on getting the source values as a string for the various webpack assets which are available through the compilation object, so there's a bit of hacky recursive function calling, but it does a job for now!

Really appreciate the prompt and helpful response on this :)

Definition:

        const addAppShellToManifest = ({ url, templates = [], regexes = [] }) => {

            return (originalManifest, compilation) => {

                /**
                 * Declaring function as we'll need to call
                 * this recursively to drill down into children
                 * for ConcatSources
                 */
                const getSourceAsString = ({ source }) => {

                    switch (source.constructor.name) {

                        case 'RawSource':
                            return source._value

                        case 'ConcatSource':
                            let output = ''
                            source.children.forEach(child => {
                                if (typeof child === 'object') {
                                    output += getSourceAsString(child)
                                } else if (typeof child === 'string') {
                                    output += child
                                } else throw `Unexpected type when attempting to get source as string: ${typeof child}`
                            })
                            return output
                    }

                }

                /**
                 * Create immutable manifest
                 */
                const manifest = originalManifest

                /**
                 * Filter out assets which do not match
                 * our provided regexes
                 */
                const sources = compilation.getAssets()
                    .filter(({ name }) => {
                        return !!regexes.find(regex => regex.test(name)) 
                    })
                    .map(getSourceAsString)

                /**
                 * String to store all our
                 * sources so we can generate
                 * a hash based on all of them
                 */
                let contents = '';

                /**
                 * Concat all sources to a single
                 * string
                 */
                for (const source of sources) {
                    contents += source
                }

                /**
                 * Concat our fixed templates to string
                 */
                for (const template of templates) {
                    contents += fs.readFileSync(template, 'utf-8');
                }

                /**
                 * Create our hash
                 */
                const md5 = crypto.createHash('md5');

                md5.update(contents);

                const revision = md5.digest('hex')

                /**
                 * Log revision for sanity check
                 */
                console.log('app shell revision', revision)

                /**
                 * Add shell to manifest and return
                 */
                manifest.push({
                    url,
                    revision
                })

                return { manifest }
            }
        }

Usage:

            return new GenerateSW({
                // ...rest of SW config
                manifestTransforms: [
                    addAppShellToManifest({
                        url: '/app-shell',
                        templates: [
                            'src/views/layouts/main.handlebars',
                        ],
                        regexes: [
                            new RegExp('^static/(js|css)/bundle.[\\S]*.(?:js|css)$'),
                        ]
                    })
                ]
            });