tup
tup copied to clipboard
Allow commands to output whole directories
It would be really neat if we could specify:
: package.json |> npm install --omit=dev --no-package-lock |> node_modules/
And for Tup to then automatically rerun npm
any time the package.json
file changes.
I would imagine that any non-generated files present in node_modules/
in that case would trigger an error, as it would have to be a generated folder. Also, other rules writing to node_modules/
or a subdirectory thereof would likely also have to trigger an error.
Just sharing as a random idea. npm
is not a particularly useful usecase; perhaps there is some way that tar
could be a useful folder-generating command, but.. hm, maybe someone else will come with a better usecase in mind? :sweat_smile:
As a long time user of both tup and npm, my reaction is, that way madness lies. So I tried it. Setting node_modules/
as an ignored output,
: package.json |> npm install --omit=dev --no-package-lock && touch dummy |> dummy ^node_modules/
Without the dummy
output, I get
[ ] 0%tup error: failed to create child process: No error
With it, I get a bunch of stuff like this:
tup error: File 'C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_051Z-debug-0.log' was written to, but is not in .tup/db. You probably should specify it as an output
-- Delete: C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_051Z-debug-0.log
tup error: File 'C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_625Z-debug-0.log' was written to, but is not in .tup/db. You probably should specify it as an output
-- Delete: C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_625Z-debug-0.log
I've had similar problems using tup with Powershell, which always wants to write to an off-tree profile directory. Perhaps npm has a "no cache" option you can use to get around this? It's not clear.
But anyway, it's not as if npm itself is deterministic under any circumstances, with or without a lockfile.
So for better or worse, tup is not really suited for the provisioning of dependencies whose files are not known a priori.
Following up...
You can get past the log problems by adding ^npm-cache/_logs
to the command output. When I try that I get:
*** tup messages ***
tup error: File 'C:/Users/gcannizzaro/npm-cache/_cacache/tmp/6c7a104c' was written to, but is not in .tup/db. You probably should specify it as an output
-- Delete: C:/Users/gcannizzaro/npm-cache/_cacache/tmp/6c7a104c
You can get past that by adding npm-cache/_cacache
or just changing the exclude to ^npm-cache/
. With that, the command succeeds.
$ tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.016s] Reading in new environment variables...
[ tup ] [0.016s] Parsing Tupfiles...
0) [0.000s] .
[ ] 100%
[ tup ] [0.016s] No files to delete.
[ tup ] [0.016s] Generating .gitignore files...
[ tup ] [0.031s] Executing Commands...
0) [2.220s] npm install --omit=dev --no-package-lock && touch dummy
up to date, audited 2 packages in 774ms
found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 8.16.0 -> 10.5.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.5.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice
[ ] 100%
[ tup ] [2.278s] Updated.
$ npm i --save-dev --no-package-lock typescript
added 1 package, and audited 3 packages in 4s
found 0 vulnerabilities
At this point I guess you do an npm install package-x
and then run tup
? The tup call is redundant at that point, but you need it to believe that it ran the command, i.e. it will do an install on the next build regardless. Or you could just hand edit package.json
and then it's not redundant. But tup doesn't know how your other commands are dependent on the contents of node_modules
, and wouldn't care if it did, because you told it to ignore.
So yeah, I don't think it's a healthy road to go down. I can say from having tried similar things.
Hm! That "ignore output" trick is a pretty handy one; didn't realize one can do that. Naturally, if the feature this issue suggests is somehow implemented, commands depending on node_modules
would have to be manually tagged as such by listing node_modules
as an order-only input.
As concerns determinism, note that you can already call commands like curl
just fine, so while that could be a road to flaky build madness, it is currently allowed by Tup.
As for the npm install
redundancy, note that you can do npm install --save --package-lock-only package-x
to get NPM to not write to node_modules
.
The main usecase for npm and npm-likes would be to have the buildsystem manage project dependencies. There are better ways to do that, however, so... I'm hoping someone might suggest a more compelling usecase for generated directories.