npmlock2nix
npmlock2nix copied to clipboard
nix-community
ENOTCACHED / cache mode is 'only-if-cached' but no cached response available
I'm not sure this isn't related to npmlock2nix, but now that my build works I get this:
[...]
building
90m..................] / rollbackFailedOptional: verb npm-session 7c5d0f9d32bc49c
> [email protected] preinstall /build/myproject
> /build/myproject/node_modules/.hooks/preinstall
npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
npm ERR! request to https://registry.npmjs.org/polka failed: cache mode is 'only-if-cached' but no cached response available.
npm ERR! A complete log of this run can be found in:
npm ERR! /build/myproject/.npm/_logs/2020-12-26T15_18_40_514Z-debug.log
builder for '/nix/store/fxhblr5b6rbzbz6mxj5y414f8b1pcz60-TODO-0.0.1.drv' failed with exit code 1
error: build of '/nix/store/fxhblr5b6rbzbz6mxj5y414f8b1pcz60-TODO-0.0.1.drv' failed
Is it possible that this is related to https://github.com/svanderburg/node2nix/issues/106 (and also https://github.com/svanderburg/node2nix/issues/70) and that this is because of bundled package-lock.json files in some packages in the dependency tree?
If this is so, I guess npmlock2nix needs to do something like https://github.com/svanderburg/node2nix/issues/106#issuecomment-442952578
I think is might be related with the other projects but neither of their solutions are likey to work here. Can you pust the relevant parts of your package.json and package-lock.json? My guess is that it is just another variant of the git references. I've multiple branches that add better/different support of them but all of them require massive cleanup.
You can reproduce this with the Sapper template: npx degit "sveltejs/sapper-template#rollup" sappertemplate
I used a vanilla npmlock2nix shell.nix like this:
{ pkgs ? import <nixpkgs> {}, npmlock2nix ? import <npmlock2nix> { inherit pkgs; } }:
npmlock2nix.shell {
src = ./.;
nodejs = pkgs.nodejs-14_x;
}
Based on the error message in the post above, it seems that polka is the culprit.
Any ideas about even a hacky/temporary workaround for this? I currently cannot use npmlock2nix for my project anymore due to this issue...
This issue is unfortunately fairly common, in my brief experience so far with npmlock2nix. The really bad bit is that it can hit you out of the blue when you add a new dependency, meaning that everything might work fine for a few weeks and then, suddenly, you've hit a a dead end.
On that note, I'm not sure whether this is the same issue or not, but if you try to set up Tailwind CSS for use with a create-react-app app as advised in the Tailwind docs here, you get the following error:
error: builder for '/nix/store/0pn16y1r4r02sfp8hfk81ca4g33j8r8p-react-nix-craco-0.1.0.drv' failed with exit code 1;
last 10 log lines:
> 90m..................] / rollbackFailedOptional: verb npm-session fa56488e4e6b8c7
> > [email protected] preinstall /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0
> > /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0/node_modules/.hooks/preinstall
> patching script interpreter paths in .
> npm ERR! code ENOTCACHEDrver: sill fetchPackage[0m
> npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
>
> npm ERR! A complete log of this run can be found in:
> npm ERR! /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0/.npm/_logs/2021-08-06T01_52_58_999Z-debug.log
>
For full logs, run 'nix log /nix/store/0pn16y1r4r02sfp8hfk81ca4g33j8r8p-react-nix-craco-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/l0dca36c585yy6y20cagbh7ll9ksfcc9-nix-shell-env.drv' failed to build
Unfortunately, I don't know how to work around this, as create-react-app requires this special compatibility release of tailwindcss.
I ran into the same error on my project, it seems to be a dependency that was pulled out of npm. If you remove your package-lock.json and recreate it (with npm install), for me it removed the problematic dependency.
unfortunately I got the same error with another dependency after that.
I may have gained some insight into the internals of this, but not much. Nag me to take some notes at some point. https://github.com/npm/pacote appears to be related (and is part of an npm checkout). Do note that this code has churned a bit between node 6 and 7.
Here's some log from where we got with peeking at the above mentioned issue today, but it doesn't seem to be the same as a related looking issue we've been fighting;
38 verbose stack Error: request to https://registry.npmjs.org/polka failed: cache mode is 'only-if-cached' but no cached response available.
38 verbose stack at /nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/make-fetch-happen/index.js:133:21
38 verbose stack at async Arborist.[nodeFromEdge] (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:934:19)
38 verbose stack at async Arborist.[buildDepStep] (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:880:11)
38 verbose stack at async Arborist.buildIdealTree (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:218:7)
38 verbose stack at async Promise.all (index 1)
38 verbose stack at async Arborist.reify (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:136:5)
38 verbose stack at async Install.install (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/lib/install.js:150:5)
This log is using a lock file generated by npm 6 so that npmlock2nix can handle it, and then we use npm 7 because the output seems easier to make sense of and use in debugging.
FYI, I have yet to encounter this with https://github.com/serokell/nix-npm-buildpackage (edit not with npm v6, anyway; see my later comment below).
Unfortunately, I have encountered this issue with nix-npm-buildpackage, at least using NPM v7.
Under the thread for https://github.com/serokell/nix-npm-buildpackage/issues/33, @yorickvP linked https://github.com/npm/cli/issues/2160, which seems to point to a fix for pacote being pushed with https://github.com/npm/pacote/commit/500a34f2f236b671fbb19a7abc8082a8a43b267f.
Based on the versions associated with nodejs/npm, this fix should be available as part of pacote 11.3.2 / nodejs 7.11.0 if I understand correctly.
Still totally lost with where that leaves us with all these projects, as they seem to be mostly up to date as far as nodejs/npm versions go.
Unfortunately, I have encountered this issue with nix-npm-buildpackage, at least using NPM v7.
Yes, I have as well with npm v7, but not with npm v6.
Hi! When I looked into it, I think my conclusion was that npm v7 now also requires the cache to exist during the npm prune step. Copying the cache into that step might help.
good news : ) should be fixed in https://github.com/nix-community/npmlock2nix/pull/93#issuecomment-911851957
good news : ) should be fixed in #93 (comment)
I tried your branch here: https://github.com/milahu/npmlock2nix/commit/3970a1b8bac6f3bd93913969c88bdc3966576cdf
I don't get ENOTCACHED anymore, but I do get this failure when trying to build a React project:
nix-build -A packages.x86_64-darwin.frontend
warning: Git tree '/Users/dhess/git/primer-frontend-web' is dirty
these 2 derivations will be built:
/nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv
/nix/store/lwsbgvhylbvw2hfpa12py680f091xvc3-primer-frontend-web-0.1.0.drv
building '/nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv'...
patching sources
configuring
no configure script, doing nothing
building
run 'npm install' ...
[ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree
error: builder for '/nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv' failed with exit code 1;
last 6 log lines:
> patching sources
> configuring
> no configure script, doing nothing
> building
> run 'npm install' ...
> [ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree
For full logs, run 'nix log /nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/lwsbgvhylbvw2hfpa12py680f091xvc3-primer-frontend-web-0.1.0.drv' failed to build
The project works with nix-npm-buildpackage.
maybe thats too silent. can you test d0c89d13f3213ffc4fe5bc275cc5b24daa67c9cc?
maybe thats too silent. can you test d0c89d1?
Sorry, no luck — back to ENOTCACHED:
nix-build -A packages.x86_64-darwin.frontend ~/git/primer-frontend-web
warning: Git tree '/Users/dhess/git/primer-frontend-web' is dirty
these 2 derivations will be built:
/nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv
/nix/store/xs4aakg1v6580sa0kxr428fvaghi58rl-primer-frontend-web-0.1.0.drv
building '/nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv'...
patching sources
configuring
no configure script, doing nothing
building
90m..................] / rollbackFailedOptional: verb npm-session 707adbebf5cc376
> [email protected] preinstall /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0
> /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/node_modules/.hooks/preinstall
patching script interpreter paths in .
npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
npm ERR! A complete log of this run can be found in:
npm ERR! /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/.npm/_logs/2021-09-03T00_46_27_051Z-debug.log
error: builder for '/nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv' failed with exit code 1;
last 10 log lines:
> 90m..................] / rollbackFailedOptional: verb npm-session 707adbebf5cc376
> > [email protected] preinstall /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0
> > /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/node_modules/.hooks/preinstall
> patching script interpreter paths in .
> npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
> npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
>
> npm ERR! A complete log of this run can be found in:
> npm ERR! /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/.npm/_logs/2021-09-03T00_46_27_051Z-debug.log
>
For full logs, run 'nix log /nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/xs4aakg1v6580sa0kxr428fvaghi58rl-primer-frontend-web-0.1.0.drv' failed to build
can you share your package.json and package-lock.json files?
Not for that particular project, but here's the one it's based on, where I first encountered this issue. With https://github.com/milahu/npmlock2nix/commit/3970a1b8bac6f3bd93913969c88bdc3966576cdf (and master), this commit works fine: https://github.com/hackworthltd/react-nix-craco-public/commit/d184c089e36f40188d9f5f6b4b493b2f736226c8
But once you try to add Tailwind & its dependencies, it breaks with the same error that I showed above: https://github.com/hackworthltd/react-nix-craco-public/commit/e4dd29ae364d0625c81ee0abfc61e96397bd7e56
im getting a broken logfile : /
loa^M[ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree^M[ ................] \ loadDep:csstype: sill fetchPackageMetaData error for tai^M[ ..............] \ loadDevDep:typescript-language-server: sill fetchPackage^M
fix: npm --progress=false, optional: npm --loglevel=verbose
now i also get
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
but the main problem here, is that requires are not patched
to debug: add cat package-lock.json | ${jq}/bin/jq
"yargs": {
"dev": true,
"integrity": "sha512-aePbxDmcYW++PaqBsJ+HYUFwCdv4LVvdnhBy78E57PIor8/OVvhMrADFFEDh8DHDFRv/O9i3lPhsENjO7QX0+A==",
"requires": {
"cliui": "^6.0.0",
"decamelize": "^1.2.0"
},
"resolved": "file:///nix/store/c4lkadnl5rqrlmr742101gcigj1106xi-yargs-15.4.1.tgz",
"version": "15.4.1"
},
intead of "cliui": "^6.0.0", it should say "cliui": "file:///nix/store/xxxxxxxxxxxx-cliui-6.0.0.tgz",
intead of
"cliui": "^6.0.0",it should say"cliui": "file:///nix/store/xxxxxxxxxxxx-cliui-6.0.0.tgz",
Hmm, are you sure that's a good solution? @andir and I discovered that Nix store paths were being written into the package-lock.json here: https://github.com/nix-community/npmlock2nix/issues/85#issuecomment-893524890 and that didn't seem like a good idea, a) because of the 211-character path length limitation (which can apparently be worked around), and b) because it wouldn't be a very useful package-lock.json file for people who aren't using Nix.
Hmm, are you sure that's a good solution?
no ... the problem started with tailwindcss which is declared as
"tailwindcss": "npm:@tailwindcss/postcss7-compat@^2.2.9",
"tailwindcss": {
"version": "npm:@tailwindcss/[email protected]",
"resolved": "https://registry.npmjs.org/@tailwindcss/postcss7-compat/-/postcss7-compat-2.2.9.tgz",
in the patched lockfile
"tailwindcss": {
"resolved": "file:///nix/store/zcggqgvhw919c209c7f4hpfq3ki6a1bw-postcss7-compat-2.2.9.tgz",
"version": "@tailwindcss/[email protected]"
},
the patched version should be 2.2.9 to make it work (workaround, maybe its a bug in npm)
echo patch lockfile for debug
#sed -i 's|"@tailwindcss/postcss7-compat@^2.2.9"|"npm:@tailwindcss/[email protected]"|' package.json
#sed -i 's|"version":"@tailwindcss/[email protected]"|"version":"npm:@tailwindcss/[email protected]"|' package-lock.json
#npm verb stack Error: request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
sed -i 's|"@tailwindcss/postcss7-compat@^2.2.9"|"2.2.9"|' package.json
sed -i 's|"version":"@tailwindcss/[email protected]"|"version":"2.2.9"|' package-lock.json
# works
echo "use npm version $(npm --version) in $(${which}/bin/which npm)"
# use npm version 6.14.14 in /nix/store/pvnvfvimzldwbx08pj53iaf3z1wmwl94-nodejs-14.17.5/bin/npm
echo "run 'npm install' ..."
npm install --offline --nodedir=${nodeSource nodejs} --progress=false # --loglevel=verbose
it wouldn't be a very useful
package-lock.jsonfile for people who aren't using Nix.
the patched json files are only used in npmlock2nix
The Tailwind version string comes straight from the documentation here: https://tailwindcss.com/docs/guides/create-react-app
It works fine with nix-npm-buildpackage.
please try 47e300362becca195b45247ce470d7f8cb42dc48
with "patching dependencies" i mean the function patchDependency in npmlock2nix/internal.nix
please try 47e3003
Closer, but still no cigar. It works fine with https://github.com/hackworthltd/react-nix-craco-public/commit/aff06f32d4befcca2a633c95a4295fec1f332d82
However, with the more complicated (private) repo I tested yesterday, it fails with this log: https://gist.github.com/dhess/084878f17db5859c0bf70f251e235731
(Edit I recreated the package-lock.json file from scratch and the error is the same.)
(Edit #2 Exactly the same problem with npm 7 (using nodejs-16_x from nixpkgs-unstable).)
probably another bug in npmlock2nix
a different version of babel-loader was detected higher up in the tree
-> why is version 8.1.0 not installed deep in /build/node_modules/react-scripts/node_modules/
possible reason: a github version github:owner/repo is replaced with wildcard version *
to debug, add this to npmlock2nix/internal.nix
cat package-lock.json | ${jq}/bin/jq -r | grep -C3 babel-loader
cat package.json | ${jq}/bin/jq -r | grep babel-loader
in the first buildPhase, before npm install --offline
add jq (for json pretty-printing) to dependencies in internal.nix line 1
then compare with your input files package-lock.json and package.json
# nix-shell -p jq
cat package-lock.json | jq -r | grep -C3 babel-loader
cat package.json | jq -r | grep babel-loader
I guess there's no real harm in making the package.json and package-lock.json files from the project available to you, which would make this a lot easier, right? :) So here you go:
package.json: https://gist.github.com/dhess/0ba93bd8685b14f88672ae8068a775b2
package-lock.json: https://gist.githubusercontent.com/dhess/1cced868803bbeda16d1b27a02216dbc/raw/346a84a241d4c2e15460df731199977d3eba4071/gistfile1.txt
could be a bug in npm 6
patched lockfile, as produced by npmlock2nix
{
"dependencies": {
"react-scripts": {
"version": "4.0.3",
"resolved": "file:///nix/store/jfkjd4w2nfzwa93q97h4db6r47m06b3a-react-scripts-4.0.3.tgz",
"dependencies": {
"babel-loader": {
"version": "8.1.0",
"resolved": "file:///nix/store/873g019wn2x550f4sdssjdl7nlgy9n3g-babel-loader-8.1.0.tgz"
},
},
"requires": {
"babel-loader": "8.1.0"
}
}
}
}
workaround in npmlock2nix:
in react-scripts.requires, replace "babel-loader": "8.1.0"
with "babel-loader": "file:///nix/store/873g019wn2x550f4sdssjdl7nlgy9n3g-babel-loader-8.1.0.tgz"
thats how i fixed the ENOTCACHED bug with my github dependencies
Thanks for your hard work on this!
I think I'll continue using nix-npm-buildpackage for the time being, since it works without any workarounds required.
weird, with nix-npm-buildpackage i also get the warning from react-scripts
this warning "should" be fixed by npm install --legacy-bundling
which disables all deduplication and creates a "deep" node_modules
i say "should" cos npm install --legacy-bundling does not work >: (
i still get a "flat" node_modules, with a global /node_modules/babel-loader
"dear npm, are my commands a joke to you?" ...
your case with "flat" node_modules
= deduplication on, note the "deduped" annotations
++ npm ls babel-loader
[email protected] /build/react-nix-craco-public
+-- @storybook/[email protected]
| `-- @storybook/[email protected]
| `-- @storybook/[email protected]
| `-- [email protected] deduped
+-- @storybook/[email protected]
| +-- @storybook/[email protected]
| | `-- @storybook/[email protected]
| | `-- @storybook/[email protected]
| | `-- [email protected] deduped
| `-- @storybook/[email protected]
| `-- [email protected]
`-- [email protected]
`-- [email protected]
"global" version of babel-loader, shared by @storybook/builder-webpack4 and @storybook/manager-webpack4
++ head node_modules/babel-loader/package.json
{
"name": "babel-loader",
"version": "8.2.2",
"local" version of babel-loader, only for react-scripts
++ head node_modules/react-scripts/node_modules/babel-loader/package.json
{
"name": "babel-loader",
"version": "8.1.0",
~~we install packages by symlinks, so we have no need for deduplication (and its side-effects)~~ nope, currently we copy packages to node_modules
maybe this can be solved with buildInputs for dependencies and nativeBuildInputs for devDependencies
we could use hardlinks like pnpm or fallback to copy if hardlinks dont work
I'm getting this issue with the wrappy package (v1.0.2). It looks like #93 was merged, so it doesn't appear to solve the issue in my case. nix-npm-buildpackage also fails with the same error.
I'm also getting the issue while trying to install gatsby-cli on aarch64-darwin with nixpkgs-22.05-darwin.
