`utils.cache.list()` should be incremental

[ehmicky]

At the moment, utils.cache.list() returns a promise resolving to an array with a list of files. When lots of files are being listed, this leads to builds hanging (see this issue). Instead, utils.cache.list() should return a stream, so the result can be incrementally read.

for (const filePath of utils.cache.list()) {
  console.log(filePath)
}

The full result could still be aggregated using the spread operator.

const filePaths = [...utils.cache.list()]

[ppannuto]

I'm not sure if this should be part of the same issue or something separate, but this interface also doesn't really match my expected behavior. All of the other APIs in the cache-utils documentation specify that they Cache a file/directory. Only the list method seems to operate only on files?

More specifically, I would expect:

var list_of_things_to_cache = [ ... ];
await utils.cache.save(list_of_things_to_cache);
var list_of_things_cached = await utils.cache.list();
// Assuming it started with an empty cache, otherwise assert the subset
assert(list_of_things_to_cache == list_of_things_cached);

I can see where it could be useful for list be more verbose, but I would imagine that as something to be opted into (i.e. a recurse or walk_directories flag) rather than the default behavior.

[ehmicky]

You're right. The solution to this problem should be to combine both:

• utils.cache.list() should not recurse (depth: 1), return both files and directories (type: 'all') and return an array (readdirp.promise())
• utils.cache.listDeep() should recurse (default depth), return only files (default type) and return an async iterator (for await (const { path } of readdirp(...)) { yield join(base, path) }).

[ehmicky]

The first part has been fixed in #1573.

The second (adding utils.cache.listDeep()) requires dropping support for Node <10.3.0 because of async iterators support. See #939.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in 1 year. It will be closed in 14 days if no further activity occurs. Thanks!

[github-actions[bot]]

This issue was closed because it had no activity for over 1 year.