Cache does not work with multiple jest workers

[imjeremyhi]

When using jest with multiple workers the .eslintcache looks like it is being overwritten by every worker. Hence adding the cache option makes no difference to performance.

Happy to try work on a PR to fix the cache overwriting issue but would need some guidance on what kind of approach would be appropriate. My initial ideas would be:

1. A separate folder which creates a cache file for every file. This prevents most of the concurrency issues but may lead to a significantly large memory overhead and lots of os reads.
2. Pass information to all spawned workers of the current run, and then lock the cache writes so each worker correctly appends to the file.
3. Or alternatively all workers create separate cache files and they get merged together in the end into a single cache file.

Do any of these ideas seem appropriate? In the meantime will do some more digging around the code base.

[SimenB]

Not sure how to best deal with this. Neither Jest nor this plugin writes to the cache directly, so maybe it makes sense to fix this in eslint? I'm not sure we can do anything here.

Eslint would need some kind of lock or something similar. write-file-atomic would kinda work, but it'd discard old writes to the cache between its own read and whenever it's able to write. A more generic lock followed read+write might be needed.

Possibly related to https://github.com/eslint/rfcs/pull/4, https://github.com/eslint/rfcs/pull/11 and https://github.com/eslint/rfcs/pull/42

---

@ljharb @rogeliog thoughts?

[ljharb]

eslint isn't designed to be run in more than one process at a time, so i'm not sure if this can really be avoided if it's eslint itself doing the cache writing.

[kirill-konshin]

Each worker may receive own cache sub-directory, but then there would be a problem to merge all caches back to one (option 3, basically).

And on subsequent runs single cache has to be provided to workers (probably just copy-paste before run).

The problem with ESLint is that it is synchronous, it only received async APIs in version 7, as mentioned here https://github.com/eslint/eslint/issues/3565, comment https://github.com/eslint/eslint/issues/3565#issuecomment-636069910.