Allow commands to output whole directories

[bojidar-bg]

It would be really neat if we could specify:

: package.json |> npm install --omit=dev --no-package-lock |> node_modules/

And for Tup to then automatically rerun npm any time the package.json file changes.

I would imagine that any non-generated files present in node_modules/ in that case would trigger an error, as it would have to be a generated folder. Also, other rules writing to node_modules/ or a subdirectory thereof would likely also have to trigger an error.

Just sharing as a random idea. npm is not a particularly useful usecase; perhaps there is some way that tar could be a useful folder-generating command, but.. hm, maybe someone else will come with a better usecase in mind? :sweat_smile:

[earthlyreason]

As a long time user of both tup and npm, my reaction is, that way madness lies. So I tried it. Setting node_modules/ as an ignored output,

: package.json |> npm install --omit=dev --no-package-lock && touch dummy |> dummy ^node_modules/

Without the dummy output, I get

 [ ]   0%tup error: failed to create child process: No error

With it, I get a bunch of stuff like this:

tup error: File 'C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_051Z-debug-0.log' was written to, but is not in .tup/db. You probably should specify it as an output
 -- Delete: C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_051Z-debug-0.log
tup error: File 'C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_625Z-debug-0.log' was written to, but is not in .tup/db. You probably should specify it as an output
 -- Delete: C:/Users/gcannizzaro/npm-cache/_logs/2024-04-01T18_53_30_625Z-debug-0.log

I've had similar problems using tup with Powershell, which always wants to write to an off-tree profile directory. Perhaps npm has a "no cache" option you can use to get around this? It's not clear.

But anyway, it's not as if npm itself is deterministic under any circumstances, with or without a lockfile.

So for better or worse, tup is not really suited for the provisioning of dependencies whose files are not known a priori.

[earthlyreason]

Following up...

You can get past the log problems by adding ^npm-cache/_logs to the command output. When I try that I get:

 *** tup messages ***
tup error: File 'C:/Users/gcannizzaro/npm-cache/_cacache/tmp/6c7a104c' was written to, but is not in .tup/db. You probably should specify it as an output
 -- Delete: C:/Users/gcannizzaro/npm-cache/_cacache/tmp/6c7a104c

You can get past that by adding npm-cache/_cacache or just changing the exclude to ^npm-cache/. With that, the command succeeds.

$ tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.016s] Reading in new environment variables...
[ tup ] [0.016s] Parsing Tupfiles...
 0) [0.000s] .
 [ ] 100%
[ tup ] [0.016s] No files to delete.
[ tup ] [0.016s] Generating .gitignore files...
[ tup ] [0.031s] Executing Commands...
 0) [2.220s] npm install --omit=dev --no-package-lock && touch dummy

up to date, audited 2 packages in 774ms

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 8.16.0 -> 10.5.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.5.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice
 [ ] 100%
[ tup ] [2.278s] Updated.
$ npm i --save-dev --no-package-lock typescript

added 1 package, and audited 3 packages in 4s

found 0 vulnerabilities

At this point I guess you do an npm install package-x and then run tup? The tup call is redundant at that point, but you need it to believe that it ran the command, i.e. it will do an install on the next build regardless. Or you could just hand edit package.json and then it's not redundant. But tup doesn't know how your other commands are dependent on the contents of node_modules, and wouldn't care if it did, because you told it to ignore.

So yeah, I don't think it's a healthy road to go down. I can say from having tried similar things.

[bojidar-bg]

Hm! That "ignore output" trick is a pretty handy one; didn't realize one can do that. Naturally, if the feature this issue suggests is somehow implemented, commands depending on node_modules would have to be manually tagged as such by listing node_modules as an order-only input.

As concerns determinism, note that you can already call commands like curl just fine, so while that could be a road to flaky build madness, it is currently allowed by Tup.

As for the npm install redundancy, note that you can do npm install --save --package-lock-only package-x to get NPM to not write to node_modules.

The main usecase for npm and npm-likes would be to have the buildsystem manage project dependencies. There are better ways to do that, however, so... I'm hoping someone might suggest a more compelling usecase for generated directories.