[question] asset & shell caching, CDN's, and server-side rendering

[tconroy]

Hi @jeffposnick! You were helping me out on StackOverflow with some SW questions, and figured this might be a better location for some longform discussion. I apologize in advanced if this is a little lengthy, I am new to service workers and trying to be thorough :)

TL;DR: I'm trying to understand how to implement the app shell caching, and could use clarity on this from the example: https://github.com/GoogleChrome/sw-precache/blob/c5e518886e1aec65d93afd32070189943dd7257e/app-shell-demo/gulpfile.babel.js#L129-L135

overview: I have an Express/React/Redux/React-Router app that makes heavy use of server-side rendering.

• app is written in ES6, transpiled with Babel and uses Webpack for build tools. I'm using the webpack-sw-precache plugin for dist builds.

• During deploy, a jenkins job takes the static assets (CSS, JS bundles) and rsyncs to a remote CDN. The server bundle (express etc.) is deployed to the server.

• Each build uniquely hashes the static assets, and a webpack.assets.json file is generated. It looks like this:

{"app":{"js":"app-85b1516dee72c0f5c025.js","css":"_all.3b6bca68cc4e0226b828.css"},"vendors":{"js":"vendors-a5f16e0ab5ebb79d95df.js"}}

• app-server-side, webpack.assets.json is loaded at startup. We use these values to dynamically build the index file.

questions:

• how do I have the service-worker cache the "app shell", when it is dynamically generated server-side? the <div id="app"><div>${html}</div></div> line in renderHtmlTemplate() is where the react markup is inserted.

should my server expose an endpoint, /shell, that simply returns an empty HTML File? ( aka what is returned in renderHtmlTemplate(), minus the ${html} bit? ), and set that as a dynamic cache in SW?

• I have API, image, and JS bundle caching working via runtimeCaching. It's unclear to me how to cache the app shell +

• how do I tell service-worker to cache my static assets, that are rsynced to the CDN? I'm using runtimeCaching for this, but it seems like I should be doing something with the staticFileGlob?

• how do I show a custom route to the user (offline screen, etc) based on the state of the service worker? It's unclear to me what needs to be implemented in my react code, vs what the service worker will handle.

---

Below are the core files involved with the server-side render:

simplified server.js: this is where the express app is set up. We serve service-worker.js and the manifest from the app server here and not a CDN ( other static assets are from CDN ).

import { matchPath } from './routes/match';

// gets served from app server, not CDN
app.get('/service-worker.js', (req, res) => {
  res.append('Content-Type', 'text/javascript');
  res.sendFile(path.resolve(__dirname, '../service-worker.js'));
});

// gets served from app server, not CDN
app.get('/manifest.json', (req, res) => {
  res.append('Content-Type', 'text/javascript');
  res.sendFile(path.resolve(__dirname, './manifest.json'));
});

app.get('*', (req, res, next) => matchPath(req, res, next));

app.listen(PORT, () => {
  console.log(`HTTP: Server listening on http://localhost:${PORT}, Ctrl+C to stop`);
});

./routes/match: this file matches the requested URL against the app routes, and populates the redux store based on it.

import React from 'react';
import fs from 'fs';
import path from 'path';
import routes from '../lib/routes';
import renderHtmlTemplate from '../templates/PageTemplate';
import { Provider } from 'react-redux';
import { createStore, applyMiddleware, compose } from 'redux';
import { createMemoryHistory, match, RouterContext } from 'react-router';
import { renderToString } from 'react-dom/server';

// this is where we load in the hashed bundle names for injecting into the 
// HTML template. on dev, we just use non-hashed names.
const webpackAssetsPath = path.resolve(__dirname, './../../webpack.assets.json');
const webpackConfig = process.env.NODE_ENV === 'production' ? 
  JSON.parse(fs.readFileSync(webpackAssetsPath, 'utf8')) : {
    app: {
      js: 'app.js',
      css: '_all.css',
    },
};

// this is some CSS we inline in the server-side response for above-the-fold assets
const aboveFoldCSS = fs.readFileSync(
  path.resolve(__dirname, '../../dist/assets/AboveFoldStream.css'),
'utf8');

// this function generates the React markup, which is then dropped into the 
// page template -- renderHtmlTemplate()
function render(res, { store, renderProps }) {
  let html = renderToString(
    <Provider store={store}>
      <RouterContext {...renderProps} />
    </Provider>
  );
  html = renderHtmlTemplate(html, { css: aboveFoldCSS, state: store.getState() });
  // send the formatted HTML to the client.
  res.send(html);
}

// performs the react-router matching 
export function matchPath(req, res, next) {
  // < lots of stuff here.. sets up memoryHistory, redux store, etc..
  // dispatches some redux actions && retrieves some async data >
  // ...

  // performs react-rouer match against the request.
  // `history` = react-router memoryHistory
  // `routes` = React-Router JSX routes
  match({ history, routes, location: req.url }, (error, redirectLocation, renderProps) => {
    if (error) {
      res.status(500).send(error.message);
      return;
    }

    if (redirectLocation) {
      res.redirect(302, redirectLocation.pathname + redirectLocation.search);
    } else if (renderProps) {
      // < some other redux dispatching occurs here to populate the initial
      // state with route-specific data >
      // ...

      // now redux state is ready, we render the output to the user
      render(res, { store, renderProps });
  });
}

PageTemplate.js: this file creates the "app shell", the markup the react app lives inside.

import serialize from 'serialize-javascript';
import * as partials from './partials';
import path from 'path';
const fs = require('fs');
const { NODE_ENV, APP_STATIC_PATH } = process.env;
const webpackAssetsPath = path.resolve(__dirname, './../../webpack.assets.json');
const manifestPath = path.resolve(__dirname, './../manifest.json');
const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
const webpackConfig = NODE_ENV === 'production'
  ? JSON.parse(fs.readFileSync(webpackAssetsPath, 'utf8'))
  : {
    app: {
      js: 'app.js',
      css: '_all.css',
    },
  };

/**
 * Returns the HTML file with app initialState embedded.
 */
export function renderHtmlTemplate(html, { css, state }) {
  return `
    <!doctype html>
    <html>
      <head>
        ${partials.staticAssets(APP_STATIC_PATH, webpackConfig, { ext: 'js', tag: 'preload' })}
        <link rel="manifest" href="/manifest.json" />
        <meta name="apple-mobile-web-app-capable" content="yes">
        <meta name="apple-mobile-web-app-title" content="${manifest.short_name}">
        ${partials.aboveFoldCSS(css)}
      </head>
      <body style="background-color: black;">
        <div id="app"><div>${html}</div></div>
        ${partials.env}
        ${partials.ssrScripts}
        <script>window.__INITIAL_STATE__ = ${serialize(state)};</script>
        ${partials.staticAssets(APP_STATIC_PATH, webpackConfig, { ext: 'js', tag: 'script' })}
        <script>
        if (window.location.protocol === 'https:' && 'serviceWorker' in navigator) {
          navigator.serviceWorker.register('/service-worker.js');
        }
        </script>
        <noscript id="deferred-styles">
          ${partials.staticAssets(APP_STATIC_PATH, webpackConfig, { ext: 'css', tag: 'link' })}
        </noscript>
      </body>
    </html>
    `;
}

export default renderHtmlTemplate;

routes.js -- these are the react routes matched against by match()

/* eslint-disable max-len */
import React from 'react';
import Route from 'react-router/lib/Route';
import ComposedAppComponent from 'components/ComposedAppComponent';
import CoverPan from 'components/container/ConnectedCoverPanComponent';
import CardPan from 'components/container/ConnectedCardPanComponent';

export function getRouteComponent(nextState, cb) {
  const { query } = nextState.location;
  if (query.c && !isNaN(query.c)) {
    cb(null, CardPan);
  } else {
    cb(null, CoverPan);
  }
}

export default (
  <Route path="/" component={ComposedAppComponent}>
    <Route path="sites/:username/:year/:month/:day/:slug(/)"
      getComponent={getRouteComponent}
    />
  </Route>
);

---

Sorry for all the questions! This is a fairly complex web app we are trying to build into a PWA, and It's just unclear to me the changes that need to be made in order to get some basic PWA functionality in place. Thank you SO MUCH for any help or guidance.

[jeffposnick]

@tconroy—apologies for not having found a chance to respond to this yet. It's still on my radar, but it's going to require a bit of investigation to familiarize myself with your setup and give you a thorough response.

[tconroy]

Thanks @jeffposnick! I look forward to your reply.

I also wanted to add ( to complicate matters further...) - most navigation in my app is dependent on a successful API call ( to retrieve data to display ) so the typical navigation flow in my app is:

• user clicks link in hamburger menu
• API request is made to retrieve destination data
• spinner is shown to the user while data is fetched
• on data success, update redux state / route / etc.

I would like to be able to handle a situation where a user does the above steps (while offline), and they get forwarded to a "Sorry, looks like you're offline (click here to refresh)".

My SW "phase 1" is to just just get the equivalent of the Chrome dinosaur "you're offline" page into my app. I'm not sure if that makes it easier or more difficult to give advice but worth mentioning. :-)

Thank you so much for your time.

[jeffposnick]

I'm not sure how comprehensive this answer is going to be, because there's a lot of different pending questions. But let's see if we can work through a few of them at a time, and maybe you can try those recommendations and then come back with a list of things that still aren't working as you expect.

Some suggestions, in no particular order:

• Add in an additional Cache-Control: max-age=0, no-cache header when serving your service-worker.js (example) to ensure your updates are fetched as soon as possible.

• Based on the contents of your PageTemplate.js, I think you're going to want to use a configuration like:

{
  dynamicUrlToDependencies: [{
    '/shell': [
      'path/to/webpack.assets.json',
      'path/to/PageTemplate.js',
      'path/to/partials.js'
    ]
  }],
  navigateFallback: '/shell',
  // ...other sw-precache config options...
}

• You should server-render requests for /shell so that it includes a <span>Loading...</span> (or the equivalent) element as the entire visible body, along with any <script> or <link> resources you need for your App Shell's JS and CSS. Don't include any dynamic content or state in the the response for /shell. It should rely on client-side rendering to populate the content.

• I believe you can use the stripPrefixMulti option to take the URLs that would be used for your assets by default and replace them with a prefix along the lines of 'https://your-cdn.com/prefix/for/assets'. You don't want to have that apply to your /shell, though, since that's not going to be served from your CDN. I honestly don't have much experience using that option, especially not for the CDN use case, as it's code that was contributed by the community.

• For your SPA navigations, which rely on runtime requests against an API, I wouldn't change anything in your client code. I'd just add in a runtimeCaching configuration with a caching strategy that you consider appropriate for the data freshness you need, and then rely on the service worker (if present) to do its thing. If you always want the freshest response when online, but are okay with the previously cached response when offline, you could try something like:

{
  runtimeCaching: [{
    urlPattern: new RegExp('/api/endpoint'),
    handler: 'networkFirst'
  }],
  // ...other sw-precache config options...
}

If there's no API response already in the cache and you're offline, then that API request will return a network error, but you need to handle that possibility anyway, since not every browser will have service worker support.

[tconroy]

Hi @jeffposnick !

Thank you so much for getting back to me. I truly appreciate it.

1 - I'm a little confused with the dynamicUrlToDependencies bit. Can you walk me through what that is doing? pageTemplate.js contains a helper function -- once invoked, it returns HTML ( as a string ) for the initial render, based on a typical react server-side flow. What is passing it into the dynamicUrlToDeps array doing with it, exactly? On my server, if I have a route to handle /shell, how does the dynamic deps tie in? Should I be building a static .html file (shell.html), that /shell on the app server serves?

2 - the shell page should load in my apps dependencies, like app.js, vendors.js, css file, etc? Should it also load in the service worker script?

3 - when client-side, is checking the window.navigator.onLine API sufficiently reliable for determining online/offline status ( and responding accordingly ie enabling/disabling non-cached links)?

4 - should /shell contain the react entrypoint so the app bootstraps? or be entirely seperate from my app code? you mentioned not including any state on /shell, but if my app by-default bootstraps its state server-side, I'm not sure how that would work?

5 - does (desktop) chrome respect the manifest.json file? I notice the start_url property. Would that essentially redirect my users to that route whenever they launch the app? I'm thinking of doing something like this:

• manifest start_url = '/shell'
• have /shell cached
• user opens app, directed to /shell, which performs a navigator.onLine check
• if online or no SW / onLine support, redirect to root /
• if offline, /shell markup shows "You're offline!" message.

would that be appropriate, or a misuse of the tooling?

thank you SO MUCH again for the help. if there's absolutely any questions you have about my setup / project config please don't hesitate to ask, I'm happy to explain anything in more details! I'm finding it very difficult to integrate this into our current project, which makes me feel I'm either missing something very obvious or our project is structured very strangely.

[hrasoa]

@jeffposnick I think the navigateFallback should be used on offline mode only. Now even on online mode the app gets pre-filed by /shell, this is bad when we already have a SSR logic for SEO.

[jeffposnick]

@hrasoa The App Shell model is compatible with SSR. You can use SSR for the initial visit to the page, and it will be used for subsequent visits from any user agents that don't have service workers.

The model works best if you can share code between the server and the client (i.e. "universal" or "isomoprhic" JavaScript). Once the SW is installed, the App Shell can be used without waiting for a response from the server, and the same code that would run server-side instead runs client-side.

See, e.g., https://www.youtube.com/watch?v=jCKZDTtUA2A and the associated project at https://github.com/GoogleChrome/sw-precache/tree/master/app-shell-demo

navigateFallback really isn't intended as a way of showing a "you're offline" page.