Web Support

[connectety]

Please add support for Flutter web. If saving infos needs be done, maybe the localStorage in webbrowsers can help.

[renefloor]

The problem is not storing the files, but that it uses sqflite to store the cache info and that doesn't support Flutter Web yet: https://github.com/tekartik/sqflite/issues/212

I'll look into this whether a switch to Moor or Hive https://github.com/renefloor/flutter_cache_manager/issues/121 is a good move. We'll have to migrate all the data, so I really have to study all the pros and cons before making a decision.

[connectety]

I looked into it a little bit more. if I understand everything correctly and did not forget any type of storage, localStorage has a maximum of 10 MB per origin and cookies have a maximum of 4 KB per domain and indexDB can get quite big but is not persistent.

In my opinion this is not enough space to cache files. But webbrowser cache files themselves quite good.

As a workaround I would propose, not caching files for web, but giving a warning and / or exception, which has to be dismissed.

[renefloor]

@connectety I'm not completely sure what you propose. Do you mean that if the platform is Web the library just uses NetworkImage instead of CachedImage? Devs will expect it to work differently.

[connectety]

use NetworkImage instead of CachedImage

Yes, for example.

Devs will expect it to work differently.

I know, and this is not the best idea I have ever had. I do not stand behind it completely, but I don't see any way to save content and a database, because the storage provided by web browsers is just too small.

Hopefully, I have forgotten some kind of storage.

And also I want to use this as a Hotfix, which can only be used after dismissing an exception or setting a flag or something. So that people will understand that it does not work as expected. This idea should still somewhat work because there is already pretty good caching in the web and browsers.

Even if this objectively bad solution gets dismissed, I wanted to point out the Problem.

[alextekartik]

I started a fork (https://github.com/alextekartik/flutter_cache_manager and https://github.com/alextekartik/flutter_cached_network_image) where I experimented caching for the web by abstracting the file system (using fs_shim) and the storage (using idb_shim). It is definitely not perfect and a better solution could be found but I think having a way to abstract the database and file system into flutter_cache_manager could allow anyone to use their preferred storage (hive, moor, sqlite, indexed db....)

But with CORS and web browser already doing some caching, the web is a different experience that could require a specific and different network caching mechanism

[IchordeDionysos]

Why not use service workers for caching? 🤔

This is what the web does really good out of the box. (In newer browser versions :D) Also with the Cache API, you can save the most files with at least 50MB storage.

This might need some refactoring of this package to use sqflite for mobile and service workers for web (maybe using the plugin_platform_interface)

Useful links: https://developer.mozilla.org/en-US/docs/Web/API/Cache

[deakjahn]

Until the support arrives, could it be made to fail silently? :-) Right now

await DefaultCacheManager().emptyCache();

never returns for me in a web app. Yes, it's easy to wrap it into an if (!kIsWeb), just saying...

[renefloor]

It doesn't cache on disk at the moment, but as @deakjahn asked it at least doesn't fail. It nicely returns the file which is just downloaded from the internet.

@IchordeDionysos I'm looking at the caching options for web. Thanks for the url

[deakjahn]

Actually, I store my files with the Cache API (not necessarily these ones but generally). Do you need some code so that you don't need to look it up yourself? Fairly straightforward.

[renefloor]

Yes it is always good to see some code, but it looks really straight forward indeed.

The main challenge I think at the moment is adapting the cache manager to separate the current storage from the web api.

[deakjahn]

I have an app-embedded platform plugin that simply offers a file storage interface for my app and gets translated to either plain old dart.io or Cache API, depending on the platform. It's tailored to my specific needs, so it's not perfect for immediate re-use without some tweaking. Basically, whenever the normal file IO speaks about folders, I use separate cache files on the web and treat a path like "foo/bar" as an item stored with the name "bar" inside a cache file named "foo". Nothing more nested than this, this was enough for my purposes.

So, the interface itself is this:

Future<bool> fileSave(String path, Uint8List contents, {String mime, bool flush = false});
Future<bool> fileSaveString(String path, String contents, {String mime, bool flush = false});
Future<Uint8List> fileRead(String path, {String mime});
Future<String> fileReadString(String path, {String mime});
Future<bool> fileExists(String path);
Future<int> fileSize(String path);
Future<bool> fileDelete(String path);
Future<bool> fileCopy(String src, String dest);
Future<bool> folderCreate(String path, {bool recursive = false});
Future<bool> folderDelete(String path, {bool recursive = false});
Future<bool> folderEmpty(String path);
Future<bool> folderExists(String path);
Future<List<String>> folderList(String path);

Not that you would need all of them, especially not the ones that store a string (basically, a text file).

[deakjahn]

The implementation is relatively simple:

import 'dart:html' as html;
import 'dart:js';
import 'dart:js_util' as util;

const String _BINARY = 'application/octet-stream';
const String _TEXT = 'text/plain';

Future<bool> fileSave<T>(String path, T contents, String mime) async {
  final parts = path.splitFirst('/');
  return util.promiseToFuture<bool>(_nativeCacheSave(parts.first, parts.last, contents, mime));
}

Future<T> fileRead<T>(String path, String mime) async {
  final parts = path.splitFirst('/');
  if (mime == _BINARY) {
    final buffer = await util.promiseToFuture<ByteBuffer>(await _nativeCacheRead(parts.first, parts.last));
    return buffer.asUint8List() as T;
  } else if (mime == _TEXT)
    return await util.promiseToFuture<String>(_nativeCacheReadString(parts.first, parts.last)) as T;
  else
    throw UnsupportedError(mime);
}

Future<bool> fileExists(String path) async {
  final parts = path.splitFirst('/');
  return util.promiseToFuture<bool>(_nativeCacheExists(parts.first, parts.last));
}

Future<int> fileSize(String path) {
  final parts = path.splitFirst('/');
  return util.promiseToFuture<int>(_nativeCacheSize(parts.first, parts.last));
}

Future<bool> fileDelete(String path) async {
  final parts = path.splitFirst('/');
  return util.promiseToFuture<bool>(_nativeCacheDelete(parts.first, parts.last));
}

Future<bool> fileCopy(String src, String dest) async {
  final parts1 = src.splitFirst('/');
  final parts2 = dest.splitFirst('/');
  return util.promiseToFuture<bool>(_nativeCacheCopy(parts1.first, parts1.last, parts2.first, parts2.last));
}

Future<bool> folderCreate(String path) async {
  final parts = path.splitFirst('/');
  await html.window.caches.open(parts.first);
  return true;
}

Future<bool> folderDelete(String path) async {
  final parts = path.splitFirst('/');
  return await html.window.caches.delete(parts.first);
}

Future<bool> folderEmpty(String path) async {
  await folderDelete(path);
  await folderCreate(path);
  return true;
}

Future<bool> folderExists(String path) async {
  final parts = path.splitFirst('/');
  return await html.window.caches.has(parts.first);
}

Future<List<String>> folderList(String path) async {
  final files = await util.promiseToFuture<List<dynamic>>(_nativeCacheList(path));
  return files.map((f) => f.toString()).toList();
}

with a simple helper:

extension Splits on String {
  List<String> splitFirst(Pattern pattern) {
    final parts = split(pattern);
    return [
      if (parts.isNotEmpty) parts.first,
      if (parts.length > 1) parts.sublist(1).join(pattern),
    ];
  }
}

[deakjahn]

The native (JavaScript, that is) implementations are:

@JS('cacheSave')
external bool _nativeCacheSave(String name, String path, dynamic contents, String mime);

@JS('cacheRead')
external JsObject _nativeCacheRead(String name, String path);

@JS('cacheReadString')
external String _nativeCacheReadString(String name, String path);

@JS('cacheCopy')
external bool _nativeCacheCopy(String name1, String src, String name2, String dest);

@JS('cacheExists')
external bool _nativeCacheExists(String name, String path);

@JS('cacheSize')
external int _nativeCacheSize(String name, String path);

@JS('cacheDelete')
external bool _nativeCacheDelete(String name, String path);

@JS('cacheList')
external JsArray _nativeCacheList(String name);

cacheSave: async function(name, path, contents, mime) {
  var cache = await window.caches.open(name);
  var blob = new Blob([contents], {type : mime});
  var options = {headers: {'Content-Type': mime, 'Content-Length': contents.length}};
  await cache.put(path, new Response(blob, options));
  return true;
},

cacheRead: async function(name, path) {
  var cache = await window.caches.open(name);
  var options = {ignoreSearch: true, ignoreMethod: true, ignoreVary: true};
  var data = await cache.match(path, options);
  return data.arrayBuffer();
},

cacheReadString: async function(name, path) {
  var cache = await window.caches.open(name);
  var options = {ignoreSearch: true, ignoreMethod: true, ignoreVary: true};
  var data = await cache.match(path, options);
  return data.text();
},

cacheCopy: async function(name1, src, name2, dest) {
  var cache1 = await window.caches.open(name1);
  var options = {ignoreSearch: true, ignoreMethod: true, ignoreVary: true};
  var data = await cache1.match(src, options);
  var cache2 = await window.caches.open(name2);
  await cache2.put(dest, data);
  return true;
},

cacheSize: async function(name, path) {
  var cache = await window.caches.open(name);
  var options = {ignoreSearch: true, ignoreMethod: true, ignoreVary: true};
  var data = await cache.match(path, options);
  return parseInt(data.headers.get('Content-Length'));
},

cacheExists: async function(name, path) {
  var cache = await window.caches.open(name);
  return await cache.match(path) != undefined;
},

cacheDelete: async function(name, path) {
  var cache = await window.caches.open(name);
  return await cache.delete(path);
},

cacheList: async function(name) {
  var cache = await window.caches.open(name);
  var files = await cache.keys();
  return files.map(f => new URL(f.url).pathname);
},

But this supposes that you already have a JS file that you can put them into. If you don't, I can also show you how I normally attach one (very much like some of my plugins, eg. https://github.com/deakjahn/flutter_dropzone, but you don't need the added complexity of a platform view, of course). Basically, the few lines starting from here, called from whatever initialization function you have handy.

[deakjahn]

Oh yes, and of course, you need to make sure you have the support:

cacheAvailable = window.caches != undefined;

You do with any sane browser you might encounter today but still... :-)

[renefloor]

But this supposes that you already have a JS file that you can put them into.

If you use dart:html in a Flutter app, isn't this automatically added?

  var blob = new Blob([contents], {type : mime});
  var options = {headers: {'Content-Type': mime, 'Content-Length': contents.length}};
  await cache.put(path, new Response(blob, options));

You basically take the content from the response and put them in a new response. Why wouldn't you just directly add the response from the server?

you need to make sure you have the support

Yes of course, thanks for that as well. Especially for older iOS devices (iOS 10 and lower) this is important.

[deakjahn]

You meant your own JS file. Flutter compiles you Dart code and puts into a JS file all right but if you want to have extra JS files to support your stuff, you have to add your own. If you have an app that you need it in, you might just get away with adding it to the Index.html file. If you have a plugin, though, you have to add it dynamically. (unless you want to instruct your users to do it manually, but that wouldn't be elegant. :-) ).

The Cache API only stores responses, not plain files. If you want to store arbitrary data, you have to create a Response whose contents is the data you need to store. But yes, in your case, you might be able to simply store the original response, yes, Note that, as I said in the beginning, I use this as a transparent cross platform storage solution, not necessarily to store actual HTTP responses.

[deakjahn]

Yes, come to think of it, you can make it quite a bit simpler than my code. The Cache API is originally conceived to store responses keyed by the actual URL. I had to bend over backwards a little bit to make this fit a more universal file storage need.

You don't need the splitFirst() stuff at all, use the URL directly and store the responses directly. The only hurdle you have to come across, really, is that you do need a supporting JS file because html.window.caches is there in Dart and you can use it directly without writing Javascript (it's a CacheStorage object), the actual cache opened with html.window.caches.open() (what would be a Cache object) has no implementation in dart:html, sadly.

At least, that would be much simpler, I think. I did see plugins that only needed one or two lines of JS executed and they created it very carefully with casting to JsObject, using callMethod() and assembling arguments tediously, but frankly, I don't think it's worth it. If you only ever need to call a single JS method, then probably yes, but for anything more complicated, it's just plain simpler and much more maintainable to do it in JavaScript directly. Not to mention that with async code like this, you would constantly need to do conversions from JS Promise to Dart Future and I don't think that would be that cheap. I preferred to do it all in JS and convert once at the end. But then again, I had my custom JS file anyway, so adding a few more functions was a no-brainer. :-)

[renefloor]

At the moment I'm working on making the cache manager more easily configurable, also for things like platform specific storage. See PR #139. It would be nice if you can help a bit with the web side of the caching.

[deakjahn]

Sure but I can't yet see where to connect. As I peek into flutter_cache_manager, I can see no specific web plugin code. Do you plan to put it somewhere here? Are some parts already written or nothing yet?

[deakjahn]

@renefloor Just to mention. There are some significant differences between 2.3.0-beta.1 and 2.3.0-rc. In release mode, the beta works flawlessly for me but the rc does not. I couldn't yet look into what exactly it is (I checked and the sources are radically different, of course, between the two).

[renefloor]

@deakjahn Are you using SKIA or normal web platform?

I'm separating the web and io file systems here: https://github.com/Baseflow/flutter_cache_manager/pull/193/files#diff-052689d714d3d840d1f79fe281252833 https://github.com/Baseflow/flutter_cache_manager/pull/193/files#diff-7b8a439543b2745e7c428f80e25e27d7

The implementation is chosen by defining which config implementation to use here: https://github.com/Baseflow/flutter_cache_manager/pull/193/files#diff-a0ac6e0ea1a2b0d4c6776f859429b621

[deakjahn]

@renefloor Both. The app normally uses Skia but when I just start it in debug mode and want to check something that's not really related to graphics rendering, that's normally the old web (only because we can't yet automatically start debug in CanvasKit mode, actually; if we could, I probably never used DomCanvas at all).

But does the caching function depend on that, really?

I'm going to set up a simple testbed app to see where we're at now. I don't want to move my current main app from that beta.1 until this is sorted out. :-)

[renefloor]

@deakjahn The caching mechanism doesn't really depend on that, but the image rendering does. The default HTML ImageElement doesn't work on Skia, so that is why (as far as I'm aware) NetworkImage doesn't work at the moment in Skia. See also: https://github.com/flutter/flutter/issues/54010 When you use Skia you should just download the image separately and render it yourself. However, when doing an HttpGet you have to manage CORS headers, so you bump into different problems.

In 2.3.0-beta I used HTML ImageElement, in 2.3.0-beta.1 I used an HTTP Get by making this package compatible. In 2.3.0-rc I added both with a flag which option to use. When you set imageRenderMethodForWeb to ImageRenderMethodForWeb. HttpGet (not the default) it should work the same as 2.3.0-beta.1.

[deakjahn]

That's confirmed. In DomCanvas, both settings work. In CanvasKit, HtmlImage fails and HttpGet works. Then I stay on the rc package.

So, where should things go forward now? Is HttpGet meant to be the way to go? Do we need to bother with the other one? Is it important to check while HtmlImage fails (I supposed you already did that, anyway, with the Flutter bug mentioned above). Or leave all that as it is and go on to caching?

[deakjahn]

Oh, I can see now why I couldn't see. The web support you mentioned is not yet in the package, it's only in a PR? Then I'll need to learn how to make that happen over here, too. :-) There is an Apply Patch in AS, so probably that's what it is. Or could it be somehow made work if I forked the package on GitHub? That would be better.

[renefloor]

The web support you mentioned is not yet in the package, it's only in a PR?

Yes, my goal was to remove all dependencies on dart:io in the web version, but that is actually really hard to do, so I'm not sure how far I will go. At least the config object is a good idea I think.

So, where should things go forward now? Is HttpGet meant to be the way to go? Do we need to bother with the other one?

For now I will support both. When using an HTML ImageElement the image always loads (no need to bother with CORS or other shit) and the caching is handled by the browser. I think this is the right way for most users, that is why it is the default.

For more power users that want more control on the caching mechanism using the HttpGet is a way with more options. In making a choice between them I follow the general discussion in the Flutter repo. Topics of interest: Use of Skia with NetworkImage: https://github.com/flutter/flutter/issues/54010 Http headers for NetworkImage: https://github.com/flutter/flutter/issues/57187 (Old) discussion about CORS: https://github.com/flutter/flutter/issues/45955

Or could it be somehow made work if I forked the package on GitHub?

You can fork from the branch and make a PR to merge it into that branch again. I guess I can also give you write access to that specific branch.

[deakjahn]

OK, I forked that one. Are there any specific internal dependencies I have to follow? The very first attempt resulted in:

Target dart2js failed: Exception: ../../flutter-beta/.pub-cache/hosted/pub.dartlang.org/cached_network_image-2.3.0-rc/lib/src/image_provider/_image_provider_web.dart:119:34:
Error: The method 'readAsBytes' isn't defined for the class 'CacheFile'.

So I probably need to adapt related modules as well? A naive

@override
Future<Uint8List> readAsBytes() {
  return _file.readAsBytes();
}

wasn't enough.

[renefloor]

@deakjahn Yes, the PR is not final yet. You can also wait till I finish more of this project.

[deakjahn]

But do you have a local copy that displays and you could zip up and send? I assume that the task at hand would be to rewrite cache_file_web.dart, to get rid of the file dependency and to read and write the Cache API here, am I correct? It's not a problem then, I guess, if other parts are still being rewritten.

[renefloor]

@deakjahn I didn't have any working version that was compatible with CachedNetworkImage. This commit fixes the build and makes sure CachedNetworkImage runs: https://github.com/Baseflow/flutter_cache_manager/pull/193/commits/8cc0dcc7c65c4f35f276555130b2d90efc7357fb

There are indeed 2 files that should be changed. file_system_web should store and retrieve files from the CacheStore. cache_file_web should not use 'package:file/file.dart' for files, but 'dart:html'. However, for that we need to make a custom implementation for methods as exists() or delete() as those files are only 'in memory' files if I'm correct.

We also need to changed the IOSink. I guess we can make a method that accepts a stream.

We also need to replace the sqflite CacheObjectProvider, but I'll do that myself.