Web Cache invalidation based on Build Number

[phackwer]

Use case

Flutter has a very strong caching done by it's service worker, which sometimes forces a user to have to clear the cache for the app even when resources are sent from the web server with a no-cache and already expired date headers. This sometimes causes problems because even with the proper code already in place in the server, the app takes a while to retrieve the latest resources from it. Sometimes, even adding ?v=(timestamp) to force this reload don't work as the cache is not controlled by the browser, but by this flutter service worker.

Proposal

Based on the version/build number of the app, cache could be deleted to force the retrieval of the latest versions from the server. This could easily and elegantly be achieve by using package info which is still not implemented as stated on https://github.com/flutter/flutter/issues/46609.

Current Successful Workaround

Currently in our pipelines we are dumping a file named version.txt in the root of the published code that contains the values from pubspec.yaml with build number updated and call caches to delete the caches everytime this build number is changed. Code is not beautiful but definitely may help those who are facing this problem:

<script>
       if ('serviceWorker' in navigator) {
        window.addEventListener('load', function () {
            //getting rid of undesired cache before running the app
            var seconds = new Date().getTime()
            xmlhttp = new XMLHttpRequest();
            xmlhttp.open("GET", '/version.txt?v=' + seconds, true);
            xmlhttp.onload = function () {
                if (xmlhttp.status == 200) {
                    var buildNumber = xmlhttp.responseText.split('+')[1];
                    var currentBuildNumber = window.localStorage.getItem('buildNumber');

                    if (currentBuildNumber && currentBuildNumber != buildNumber) {
                        caches.delete('flutter-app-manifest');
                        caches.delete('flutter-temp-cache');
                        caches.delete('flutter-app-cache');
                    }

                    window.localStorage.setItem('buildNumber', buildNumber);
                }
                navigator.serviceWorker.register('flutter_service_worker.js');
            }
            xmlhttp.error = function () {navigator.serviceWorker.register('flutter_service_worker.js');}
            xmlhttp.abort = function () {navigator.serviceWorker.register('flutter_service_worker.js');}
            xmlhttp.timeout = function () {navigator.serviceWorker.register('flutter_service_worker.js');}
            xmlhttp.send();
        });
    }
</script>

[jlubeck]

@phackwer Just wanted to thank you for the workaround and also ask you if with this script do I also need to add the ?v= to <script src="main.dart.js?v=" type="application/javascript"></script> ?

[yjbanov]

/cc @jonahwilliams What are your thoughts on this?

[jonahwilliams]

I think we should generally weaken the caching to online-first, with cache busting query strings on the app shell. If apps are still not being updated after that change, it is probably due to a service misconfiguration that we can't work around

[coleweinman]

The strong caching is such a big problem. The new service works just stay not activated until the cache is manually cleared which results in old versions of apps running indefinitely. I hope this can be fixed in Flutter soon, difficult to have live web apps with this problem.

[phackwer]

@phackwer Just wanted to thank you for the workaround and also ask you if with this script do I also need to add the ?v= to <script src="main.dart.js?v=" type="application/javascript"></script> ?

Hi, @jlubeck I still have this too. :-) My pipelines do a dump of the buildnumber to a version.txt file. Headers from nginx try to avoid having the browser to cache it, but since we call this file to present version in flutter it was getting cached there. Only way was to call the version.txt file in the index.html before flutter service and force an invalidation of flutter caches when the local storage in the browser had a different buildnumber stored there.

[TheCGDF]

@phackwer Just wanted to thank you for the workaround and also ask you if with this script do I also need to add the ?v= to <script src="main.dart.js?v=" type="application/javascript"></script> ?

But index.html will be cached.

[phackwer]

set header to index.html not to cache on nginx or apache. I've also set the headers for version.txt not to be cached, so this kind of solves the problem. Sometimes however I still need to press reload 2 times. Don't know why.

We use bitbucket pipelines (yaml) so replace the command is this

• &tagBuildNumberOnMainDart echo "Tagging index.html" && sed -i -e ''s,main.dart.js,main.dart.js?v=${BITBUCKET_BUILD_NUMBER},g'' src/build/web/index.html && cat src/build/web/index.html

Because it's a YAML file, the sed had 2 single quotes, and not a double quote. So, cleaned up command would be

sed -i -e 's,main.dart.js,main.dart.js?v=${BITBUCKET_BUILD_NUMBER},g' src/build/web/index.html && cat src/build/web/index.html

Where BITBUCKET_BUILD_NUMBER would be whatever you want to append to it. Run this after you finished your build and before you copy the built code to docker or zip or whatever other means you use to deploy it.

[TheCGDF]

We are using CDN so that we cannot control the final header. CDN will always dismiss our Cache-Control in header.

[phackwer]

If you are using AWS as CDN you can setup the file to be sent with expired headers. Check with the service you use and make this file to be sent always as expired. All services support this kind of feature

https://www.google.com/search?q=setting+cdn+header+to+expire+a+specific+file&oq=setting+cdn+header+to+expire+a+specific+file&aqs=chrome..69i57.17706j0j1&sourceid=chrome&ie=UTF-8

[phackwer]

Here is an interesting post about it:

https://www.keycdn.com/support/http-caching-headers#exclude-specific-assets-from-being-cached-using-nginx

[phackwer]

Another for Azure

https://docs.microsoft.com/en-us/azure/cdn/cdn-manage-expiration-of-cloud-service-content

So, check with the service you use.

[oantajames]

Any news regarding this issue? Currently a big pain for us. Thanks!

[venkatd]

Yes would love to have some movement on this issue. We're currently doing our own workaround which renames our main.dart.js file but we lose some of the caching benefits.

I'm not sure of the technical implications, but our dev experience is:

• Customer has a bug
• We merge in a hotfix and it gets deployed to netlify
• We ask customer to reload the page
• Customer still has the bug
• We ask customer to clear their cache and reload
• :(

The snippet of our netlify build script is here:

  async onBuild({ constants }) {
    const fingerprint = process.env.COMMIT_REF
    const fingerprintedFileName = `main-${fingerprint}.dart.js`

    const htmlPath = path.join(constants.PUBLISH_DIR, 'index.html')
    const jsPath = path.join(constants.PUBLISH_DIR, 'main.dart.js')
    const flutterServiceWorkerPath = path.join(constants.PUBLISH_DIR, 'flutter_service_worker.js')

    const newJsPath = path.join(constants.PUBLISH_DIR, fingerprintedFileName)

    await rename(jsPath, newJsPath)

    console.log({ fingerprintedFileName, htmlPath, jsPath, newJsPath })

    await replaceInFile(htmlPath, (html) => {
      return html.replace(/main\.dart\.js/g, fingerprintedFileName);
    })

    await replaceInFile(flutterServiceWorkerPath, (js) => {
      return js.replace(/main\.dart\.js/g, fingerprintedFileName);
    })
  },

So we replace all occurrences of main.dart.js with main-commit.dart.js to force a cache invalidation

[mikeesouth]

Any update here? I'm using ?v= on my main.dart.js and other files referenced in index.html and that is all working just fine. I also write the build number to local storage and I have it available in my app. But Flutter assets are still not invalidated. So I tried the workaround suggested by @phackwer in the first post and I'm pretty sure that worked just fine about a month ago or so (on Flutter beta). But now - it does not work anymore. I'm using the JS code in "Current Successul Workaround" and I have ensured that my app calls these commands when a new build is available:

caches.delete('flutter-app-manifest');
caches.delete('flutter-temp-cache');
caches.delete('flutter-app-cache');

But even with these commands, the browser still shows my .ttf files, my .json files and other assets as being read from (disk cache) and not updated. The result is that my translated texts from json files are missing and my font files aren't updated.

Any tips on how to cache bust Flutter assets successfully? Cache busting on files served from the server is not a problem and the main.dart.js file for the flutter app also breaks the cache correctly but the flutter assets within the app are cached.

I'm using Flutter beta 1.25.0-8.3.pre and I cannot use flutter beta 1.26.x yet since it has a breaking bug with SVGs when using SKIA.

[azbuky]

@mikeesouth you are deleting the service worker's cache but depending on the Cache-Control headers sent by the server, the browser can also cache the files in it own caches (which seems the case with your assets). You should set the cache response directive on your server to Cache-Control: no-cache so the browser will validate the requests with the server.

[mikeesouth]

@azbuky thanks for the tip, I will try no-cache but I do have Cache-Control headers in place already with an expire of 1 hour (3600 seconds). But Chrome are still fetching my Flutter assets from disk cache, even after 1 hour and after 10 hours and still after 24 hours, is there an explanation to this? I will try no-cache but that's a bit harsh too since ever request will fetch the resources, right? Or will the flutter service workers cache be used in that case?

I also note that I do not have an expire header on my static resources from the server, is that necessary in addition to Cache-Control?

I specify the font file in my flutter app yaml file, like this:

fonts:
  - asset: assets/fonts/IconFont.ttf

And I use it as fontFamily: 'IconFont' so nothing special in how I load the font in my Flutter code.

The request for my font file is like this:

**General**
Request URL: https://example.com/assets/assets/fonts/IconFont.ttf
Request Method: GET
Status Code: 200 OK (from disk cache)
Remote Address: 1.2.3.4:443
Referrer Policy: strict-origin-when-cross-origin

**Response headers**
Accept-Ranges: bytes
Cache-Control: public,maxAge=3600
Content-Length: 7200
Content-Type: application/x-font-ttf
Date: Tue, 09 Feb 2021 13:25:45 GMT
ETag: "1d6ed80e5bd7a20"
Last-Modified: Mon, 18 Jan 2021 10:01:32 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET

**Request headers**
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,sv;q=0.8
Connection: keep-alive
Cookie: <REMOVED>
Host: example.com
Referer: https://example.com/
sec-ch-ua: "Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36

[azbuky]

@mikeesouth no-cache does not prevent the browser from caching the resource, it will just always validate with the server before using it. Also after the service worker caches the new resources it will serve future requests from its cache.

Regarding your example it looks like the Last-Modified header has an 18th Jan date so maybe this is why the browser uses the file from it's cache even though max age might have expired.

I think the main confusion with caching is due to the fact that there are two caches between the app and the server. One is managed by the service worker (the one that you clear with the "Current Successful Workaround"), the other one is the browser's HTTP cache that can be controlled using the Cache-Control headers set by the server (or it can be controlled by the service worker by modifying the fetch request).

There is #75535 that should be merged soon and fix the issues with cache invalidation.

[mikeesouth]

@azbuky Hmm, ok. Thanks for the info, I did not know that about no-cache, i.e. that it validates with the server. I tried with no-cache and Expire: -1 and that fetched all the files, every time. So my payload for the app went up from ~12 kb to ~2.4 mb. But with only no-cache (not sending Expire at all) it does indeed send a request to the server and validates the file, returns HTTP Status 304 "Not Modified" and each of those requests are 273 bytes. So with no-cache my payload went up from ~12 kb to ~16 kb which is more than OK in my case.

This makes me realise that I have no clue on how Cache-Control actually works but I will read up on that, some rainy day :)

Thanks for the #75535 tip, I'm following it now. I guess the service-worker workaround in the first post is still needed until #75535 is fixed even if I have no-cache in the response from my server?

[pablo-threadable]

Problem about the full no-cache is that you WANT some cache. But since it's impossible to control properly for how long to cache (even with 6 hours cache only you could still have users looking at an old code for 6 hours) an invalidation based on a dumped version.json/txt file that is never cached, should be enough to avoid problems.

[joceljunior]

Sim, adoraria ter algum movimento sobre esta questão. No momento, estamos fazendo nossa própria solução alternativa que renomeia nosso main.dart.jsarquivo, mas perdemos alguns dos benefícios do cache.

Não tenho certeza das implicações técnicas, mas nossa experiência de desenvolvimento é:

• O cliente tem um bug
• Nós mesclamos em um hotfix e ele é implantado no netlify
• Pedimos ao cliente que recarregue a página
• Cliente ainda tem o bug
• Pedimos ao cliente que limpe seu cache e recarregue
• :(

O trecho do nosso script de compilação netlify está aqui:

  async  onBuild ({ constantes }) {
     const fingerprint = process.env. COMMIT_REF 
    const fingerprintedFileName = `main - ${fingerprint}.dart.js`

    const htmlPath = caminho. join (constantes. PUBLISH_DIR , 'index.html' )
     const jsPath = path. join (constantes. PUBLISH_DIR , 'main.dart.js' )
     const flutterServiceWorkerPath = path. join (constantes. PUBLISH_DIR , 'flutter_service_worker.js' )

    const newJsPath = caminho. join (constantes. PUBLISH_DIR , fingerprintedFileName)

    aguardar  renomear (jsPath, newJsPath)

    console. log ({ fingerprintedFileName, htmlPath, jsPath, newJsPath })

    await  replaceInFile (htmlPath, (html) => {
       return html. replace ( / main\.dart\.js / g, fingerprintedFileName);
    })

    await  replaceInFile (flutterServiceWorkerPath, (js) => {
       return js. replace ( / main\.dart\.js / g, fingerprintedFileName);
    })
  },

Então, substituímos todas as ocorrências de main.dart.jspor main-commit.dart.jspara forçar uma invalidação de cache

Did rename or mais.dart.js work for voice?

[kaminoan-dev]

Is the workaround in the original post still the valid solution?

@azbuky, were you able to "merge" this solution with the new service worker mechanism as you seem to indicate in https://github.com/flutter/flutter/pull/75535#issuecomment-775551128? Or what you mean is that once that fix is merged into the Flutter stable branch this workaround will no longer be needed?

[Macacoazul01]

@phackwer any update on this or workaround for flutter 3.3?

[Macacoazul01]

We are having to tell our clients to clear the cache every update...

[treeder]

Did you create your flutter web app prior to flutter 2.10? If so, you are using the old loader and I think they did some tricks with the new loading scripts in index.html that make this less of a problem.

See this for regenerating the index.html file: https://docs.flutter.dev/development/platform-integration/web/initialization#upgrading-an-older-project

[Macacoazul01]

Gonna try later today and see if it helps @treeder. Tks

[pablo-threadable]

Hi. I'm not sure, I'll check later. Currently I don't have any new projects with web to try it.

On Fri, 23 Sept 2022, 13:28 Gian, @.***> wrote:

@phackwer https://github.com/phackwer any update on this or workaround for flutter 3.3?

— Reply to this email directly, view it on GitHub https://github.com/flutter/flutter/issues/63500#issuecomment-1256147710, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZQABGWM3UMV77OH5A4QCLV7WO5TANCNFSM4P3H65KA . You are receiving this because you commented.Message ID: @.***>

[LenWdk]

With Flutter 3.3.3 serviceWorkerVersion seems to fix this

[Macacoazul01]

@LenWdk is there any setting that we need to edit or is an automatic fix?

[LenWdk]

I guess you need to run flutter create . again to create the latest index.html. It should contain var serviceWorkerVersion = null;. flutter build will set that value to a new hash each time you run it. That solved the web cache issue for us.

[sgehrman]

I updated my index.html to the latest, but it still doesn't work. Flutter 3.3.7