cacheable
cacheable copied to clipboard
jaredwray
Portable cache files
Is your feature request related to a problem? Please describe.
We would like to make ESLint cache files portable.
Use case 1: The cache file is created/updated in CI. The goal is to make the cache file reusable for CI runs, but different CI runs may have different directory structures. Use case 2: The cache file is checked in the source control and shared among developers who can update it and commit the changes. Developers typically have different directory structures on local machines.
By "directory structures" I mean the location where the project is stored locally. For example, on one machine it can be /projects/my-app, on another one /work/my-app, /projects/work-my-app or C:\projects\my-app.
References:
- https://github.com/eslint/rfcs/pull/114
- https://github.com/eslint/eslint/issues/16493
- https://github.com/eslint/eslint/issues/16741
Describe the solution you'd like
I think the best solution would be to store file paths relative to the location of the cache file. I see the latest file-entry-cache v10.0.4 can store relative paths in the cache file, but not sure if and how it would be possible to make them relative to the location of the cache file.
Regarding the solution with renameAbsolutePathKeys suggested by @jaredwray in https://github.com/eslint/rfcs/pull/114#issuecomment-2394777170, I think it isn't applicable for these use cases because we don't know oldPath, and in the Use case 2 there can be multiple old paths.
@mdjermanovic - thanks. relative pathing should work and I will add in more tests around it but you can use it no problem and it should just work. Is there a scenario where the new version doesnt?
@mdjermanovic - I added more tests around this but you should be ok using relative or absolute paths. I am going to make it even easier in the next big update that should be out this week where you will not be required to set the current working directory or the property using relative paths.
👋 I'm also interested in this, I've tried to hack my way around the absolute path requirement for the cache in ESLint v8 with little luck. If I manually update the path to resolve to the correct path in CI, I still get cache misses which I think are related to the use of mtime by this library (https://github.com/jaredwray/cacheable/blob/main/packages/file-entry-cache/src/index.ts#L299-L301). It looks like that reflects git clone time and not file touch time, https://stackoverflow.com/questions/21735435/git-clone-changes-file-modification-time. Would that also make sense to disable for the portable cache files?
@aramissennyeydd - we are working on this now but the relative path storing is going to take a bit longer to get right. We just release the option to select if you want to use the modified time.
https://github.com/jaredwray/cacheable/issues/1047
@aramissennyeydd @mdjermanovic - this should get released in the next couple days. It will be a breaking change as we are removing currentWorkingDirectory option (I don't see that you are using this). What is the best way we can test this and make sure the update makes sense?
Do you want me to publish it as a beta on npm? How can we make sure this does what we need?
@mdjermanovic and @aramissennyeydd - let me know how I can help test this.
@aramissennyeydd
👋 I'm also interested in this, I've tried to hack my way around the absolute path requirement for the cache in ESLint v8 with little luck. If I manually update the path to resolve to the correct path in CI, I still get cache misses which I think are related to the use of mtime by this library (https://github.com/jaredwray/cacheable/blob/main/packages/file-entry-cache/src/index.ts#L299-L301). It looks like that reflects git clone time and not file touch time, https://stackoverflow.com/questions/21735435/git-clone-changes-file-modification-time. Would that also make sense to disable for the portable cache files?
You can use --cache-strategy content ESLint CLI option.
https://eslint.org/docs/latest/use/command-line-interface#--cache-strategy
You can test it by doing
npm install [email protected]Thanks! I'll take a look.
@mdjermanovic - let me know how it looks and if you need any changes as we plan to release in the next couple weeks.
@jaredwray are relative paths expected to be relative to the current working directory? I'm getting ENOENT when trying to pass a path relative to the cache file.
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const useChecksum = true;
const cache = fileEntryCache.create(
'.eslintcache',
path.resolve(__dirname, 'cache'),
useChecksum
);
const filePath = '../src/a.js';
const fileDescriptor = cache.getFileDescriptor(filePath);
console.log(fileDescriptor);
cache.reconcile();
Here's a StackBlitz repro: https://stackblitz.com/edit/stackblitz-starters-bapiunry?file=init-relative.js
@mdjermanovic - I created a new version and it is live now in beta
npm install file-entry-cache@beta
This now has cwd option so you can set the current working directory and be able to move the files around easier. https://www.npmjs.com/package/file-entry-cache/v/11.0.0-beta.2#path-handling-and-current-working-directory
By default we do use process.cwd() for the cwd so this should help with your example above.
@mdjermanovic - I have updated file-entry-cache@beta so make sure you update it in your code when testing. I also noticed that https://stackblitz.com/edit/stackblitz-starters-bapiunry?file=init-relative.js you should change this to work:
const filePath = './src/a.js';
The issue is that you were going back past the root of the project. Finally, I have added a new test file that shows how you can copy files to a new folder and rename a folder and it still tracks that the file inside does not change:
https://github.com/jaredwray/cacheable/blob/main/packages/file-entry-cache/test/relative-eslint.test.ts
test("relative pathing works on files and cache", () => {
// create the unique cache so there are no conflicts
const cacheDirectory = ".cache";
const file = "./index.ts";
const useCheckSum = true;
const testFixturesPath = "./test/fixtures-relative";
const testFixturesPathRename = "./test/fixtures-relative-foo";
const cache = fileEntryCache.create(
".eslintcache-foo43",
cacheDirectory,
useCheckSum,
path.resolve("./src"),
);
const indexDescriptor1 = cache.getFileDescriptor(file);
expect(indexDescriptor1.changed).toBe(true);
cache.reconcile();
// validate that it didnt change
const indexDescriptor2 = cache.getFileDescriptor(file);
expect(indexDescriptor2.changed).toBe(false);
expect(indexDescriptor1.meta.hash).toEqual(indexDescriptor2.meta.hash);
// copy the file into a temp directory to show a move
fs.cpSync("./src/index.ts", `${testFixturesPath}/index.ts`, {
force: true,
});
// update the cwd path
cache.cwd = path.resolve(testFixturesPath);
const indexDescriptor3 = cache.getFileDescriptor(file);
// validate that the file via hash is not different
expect(indexDescriptor3.changed).toBe(false);
// rename the fixtures path
fs.renameSync(testFixturesPath, testFixturesPathRename);
// change the cwd again
cache.cwd = path.resolve(testFixturesPathRename);
// get the file again in the new current working directory
const indexDescriptor4 = cache.getFileDescriptor(file);
// validate a final time that if renamed folder but file is same it is good
expect(indexDescriptor4.changed).toBe(false);
expect(indexDescriptor4.meta.hash).toEqual(indexDescriptor1.meta.hash);
// clean up
fs.rmSync(path.resolve(cacheDirectory), {
recursive: true,
force: true,
});
fs.rmSync(path.resolve(testFixturesPath), {
recursive: true,
force: true,
});
fs.rmSync(path.resolve(testFixturesPathRename), {
recursive: true,
force: true,
});
});
@jaredwray thanks for the update!
I still can't make paths being relative to the location of the cache file (i.e., to the cache directory).
Here is what I was trying:
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const cacheFilename = ".eslintcache";
const cacheDirectory = path.resolve(__dirname, "cache");
const useChecksum = true;
const cwd = cacheDirectory;
const cache = fileEntryCache.create(
cacheFilename,
cacheDirectory,
useChecksum,
cwd
);
const filePath = "../src/a.js"; // relative to the "cache" dir
const fileDescriptor = cache.getFileDescriptor(filePath);
fileDescriptor.meta.data = { foo: "bar" };
cache.reconcile();
But it results in an error:
Error: Path traversal attempt blocked: "../src/a.js" resolves outside of working directory "C:\projects\tmp\tmp\cache"
at FileEntryCache.getAbsolutePath (C:\projects\tmp\tmp\node_modules\file-entry-cache\dist\index.cjs:405:17)
at FileEntryCache.getFileDescriptor (C:\projects\tmp\tmp\node_modules\file-entry-cache\dist\index.cjs:261:31)
at Object.<anonymous> (C:\projects\tmp\tmp\write.js:19:30)
at Module._compile (node:internal/modules/cjs/loader:1554:14)
at Object..js (node:internal/modules/cjs/loader:1706:10)
at Module.load (node:internal/modules/cjs/loader:1289:32)
at Function._load (node:internal/modules/cjs/loader:1108:12)
at TracingChannel.traceSync (node:diagnostics_channel:322:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:220:24)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:170:5)
Node.js v22.14.0
@mdjermanovic - stackblitz is causing that error. you need to change:
const filePath = "./src/a.js"; // relative to the "cache" dir
const filePath = "./src/a.js"; // relative to the "cache" dir
I tried locally with this change, but it's creating an empty cache file (the content of the cache file is [[]]).
// write.js
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const cacheFilename = ".eslintcache";
const cacheDirectory = path.resolve(__dirname, "cache");
const useChecksum = true;
const cwd = cacheDirectory;
const cache = fileEntryCache.create(
cacheFilename,
cacheDirectory,
useChecksum,
cwd
);
const filePath = "./src/a.js"; // relative to the "cache" dir
const fileDescriptor = cache.getFileDescriptor(filePath);
fileDescriptor.meta.data = { foo: "bar" };
cache.reconcile();
To clarify, here's the file and directory structure:
my-project/
├── cache/
│ └── .eslintcache
└── src/
└── a.js
I was first trying to reference src/a.js as relative to cache/.eslintcache (as suggested in https://github.com/eslint/rfcs/pull/114#discussion_r1514962901), hence ../src/a.js. The idea is that relative file paths stored in the cache file don't depend on the current work directory, but only on the location of the cache file.
I updated the example you had before to show the code that works: https://stackblitz.com/edit/stackblitz-starters-k9c4xgvu?file=init-relative.js
Also, Here is an example with cwd being used. I dont think you should base it off of the cache directory as it is really about understanding the relative paths if they move, correct?
https://stackblitz.com/edit/stackblitz-starters-qahyatkx?file=package.json,write.js
@mdjermanovic - I have implemented the meta changes so you can have other keys on it. 🎉
Also, I implemented a logger so that you can run trace on it to get more details while testing:
const logger = pino({
level: 'trace'
});
// Pass logger in constructor
const cache = new fileEntryCache.FileEntryCache({
logger,
cacheId: 'my-cache'
});
@mdjermanovic - with all the changes are in for 11.0.0-beta.4: https://www.npmjs.com/package/file-entry-cache/v/11.0.0-beta.4
I dont think you should base it off of the cache directory as it is really about understanding the relative paths if they move, correct?
Yes, the motivation for using relative paths is that the files tracked in the cache file may have different absolute paths on different machines where the project is cloned. So, by storing relative paths, we can have the same cache file being usable on different machines.
But, there's a question of which path should be used as the base for calculating relative paths. Since ESLint can be run from different directories within the project, and in ESLint there's no concept of "root" directory for a project, we came to the conclusion that storing paths relative to the cache file would be the best solution, as they would always resolve the same regardless of the directory from which ESLint is run.
For example:
$ npx eslint src/a.js --cache --cache-location cache/.eslintcache
$ cd src
$ npx eslint a.js --cache --cache-location ../cache/.eslintcache
Both commands should retrieve the same entry from the cache file, but they are run from different directories. This already works well when we're passing absolute paths. With relative paths, if we pass the current working directory as cwd, and filePath relative to the current directory, they wouldn't refer to the same entry in the cache file:
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const cacheFilename = '.eslintcache';
const cacheDirectory = path.resolve(__dirname, 'cache');
const useChecksum = true;
//
// write
//
const cwd = __dirname;
const cache = fileEntryCache.create(
cacheFilename,
cacheDirectory,
useChecksum,
cwd
);
const filePath = 'src/a.js';
const fileDescriptor = cache.getFileDescriptor(filePath);
fileDescriptor.meta.data = { foo: 'bar' };
cache.reconcile();
//
// read attempt 1
//
const cwd1 = __dirname;
const cache1 = fileEntryCache.create(
cacheFilename,
cacheDirectory,
useChecksum,
cwd1
);
const filePath1 = 'src/a.js';
const fileDescriptor1 = cache1.getFileDescriptor(filePath1);
console.log(fileDescriptor1); // changed: false (cache hit)
//
// read attempt 2
//
const cwd2 = path.resolve(__dirname, "src"); // now we are in the `src` directory
const cache2 = fileEntryCache.create(
cacheFilename,
cacheDirectory,
useChecksum,
cwd2
);
const filePath2 = 'a.js';
const fileDescriptor2 = cache2.getFileDescriptor(filePath2);
console.log(fileDescriptor2); // changed: true (cache miss)
@mdjermanovic - thanks so much and I have made some changes to help support this with 11.0.0-beta.5:
strictPaths - Because of security concerns we added the error that you are seeing where you cannot get out of. I have made the default behavior of this to be false.
useAbsolutePathAsKey - this has been added as with the scenario above we need to keep the same key no matter what directory you are in. You will need to set this to true as by default it is false.
create() and createFromFile - since we added more options the function was getting pretty long. To resolve this I added a new type called CreateOptions where you can set most of the options.
here is a working example:
const cacheDirectory = "./.cache";
const cacheId = ".eslintcache-202121212";
const file = "../src/index.ts";
const useCheckSum = true;
const useAbsolutePathAsKey = true;
const cwd = cacheDirectory;
const options: CreateOptions = {
useCheckSum,
useAbsolutePathAsKey,
cwd,
};
const cache = fileEntryCache.create(cacheId, cacheDirectory, options);
const fileDescriptor = cache.getFileDescriptor(file);
expect(fileDescriptor.changed).toBe(true);
expect(fileDescriptor.meta.hash).toBeDefined();
cache.reconcile();
const fileDescriptor2 = cache.getFileDescriptor(file);
expect(fileDescriptor2.changed).toBe(false);
expect(fileDescriptor2.meta.hash).toBeDefined();
@jaredwray - with useAbsolutePathAsKey: true, files would be stored in the cache with absolute paths so we'd lose portability?
We could always calculate paths relative to the cache file. I was able to achieve what we want without useAbsolutePathAsKey: true, this way:
// write.js
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const cacheFilename = '.eslintcache';
const cacheDirectory = path.resolve(__dirname, 'cache');
const cache = fileEntryCache.create(
cacheFilename,
cacheDirectory,
{
useCheckSum: true,
cwd: cacheDirectory,
}
);
const filePath = '../src/a.js'; // relative to the `cache/.eslintcache` file
const fileDescriptor = cache.getFileDescriptor(filePath);
fileDescriptor.meta.data = { foo: 'bar' };
cache.reconcile();
[["1"],{"key":"2","value":"3"},"../src/a.js",{"size":3,"mtime":1760100678225,"hash":"4","data":"5"},"acbd18db4cc2f85cedef654fccc4a4d8",{"foo":"6"},"bar"]
// read.js
const path = require('node:path');
const fileEntryCache = require('file-entry-cache');
const cacheFilename = '.eslintcache';
const cacheDirectory = path.resolve(__dirname, 'cache');
const cache = fileEntryCache.create(
cacheFilename,
cacheDirectory,
{
useCheckSum: true,
cwd: cacheDirectory,
}
);
const filePath = '../src/a.js'; // relative to the `cache/.eslintcache` file
const fileDescriptor = cache.getFileDescriptor(filePath);
console.log(fileDescriptor);
/*
{
key: '../src/a.js',
changed: false,
meta: {
size: 3,
mtime: 1760100678225,
hash: 'acbd18db4cc2f85cedef654fccc4a4d8',
data: { foo: 'bar' }
}
}
*/
Reading from the cache file works well when the cache file is moved to another directory along with the files it tracks, so I think this usage solves our use case.
@mdjermanovic - yep. both ways will work. Can you validate everything before I release v11?
After more testing, I believe that file-entry-cache 11.0.0-beta.5 provides features to make cache files portable, and that it supports https://github.com/eslint/rfcs/pull/114 👍
@mdjermanovic - this has now been released as 11.0.0. Thanks so much for your patience.