mod/modcache: consider making all files read-only, including zip and mod files

[jpluscplusm]

trafficstars

What version of CUE are you using (cue version)?

$ cue version
cue version v0.8.1

go version go1.22.1
      -buildmode exe
       -compiler gc
       -trimpath true
     CGO_ENABLED 0
          GOARCH amd64
            GOOS linux
         GOAMD64 v1

Does this issue reproduce with the latest stable release?

Given the small and unrelated delta between 0.8.1 and 0.8.2, I assume that this issue would still strike.

What did you do?

(A repro script would be a little too fiddly for this, so please forgive the narrative bug report)

With the modules experiment enabled, cue mod tidy fetches and unpacks module archives:

$ cat main.cue
package frostyapp

import "github.com/cue-labs/examples/frostyconfig@v0"

config: frostyconfig.#Config & {
	appName: "alpha"
	port:    80
	features: logging: true
}
$ cue mod init cue.example
$ cue mod tidy

The unpacked modules under ~/.cache/cue/<domain>/ are created without the umask that's set in my environment (0022):

$ umask
0022
$ find ~/.cache/cue/ -type f -ls 
 48155501      0 -rw-r--r--   1 user user        0 Apr 29 14:43 /home/user/.cache/cue/cache/download/github.com/cue-labs/examples/frostyconfig/@v/v0.0.1.lock
 48155502      4 -rw-r--r--   1 user user      601 Apr 29 14:43 /home/user/.cache/cue/cache/download/github.com/cue-labs/examples/frostyconfig/@v/v0.0.1.zip
 48155511      4 -rw-r--r--   1 user user       94 Apr 29 14:43 /home/user/.cache/cue/cache/download/github.com/cue-labs/examples/frostyconfig/@v/v0.0.1.mod
 48155508      4 -r--r--r--   1 user user      482 Apr 29 14:43 /home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/config.cue
 48155510      4 -r--r--r--   1 user user       94 Apr 29 14:43 /home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/cue.mod/module.cue

This renders the unpacked files annoying to clean up, should I wish to do so, despite them being stored on my machine:

$ rm -rf ~/.cache/cue/*
rm: cannot remove '/home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/config.cue': Permission denied
rm: cannot remove '/home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/cue.mod/module.cue': Permission denied

I am, of course, able to fix this. But I shouldn't have to.

What did you expect to see?

I expected files created by the cue command to obey the umask setting of the environment in which the command was invoked.

What did you see instead?

Files created with restrictive permissions.

[jpluscplusm]

As a data point, on a long-running Linux machine, with plenty of programs storing their ephemeral/etc data under ~/.cache/, here's a summary of the files stored under there, solely looking at the user-read and user-write permissions:

$ find ~/.cache/ -type f -ls | cut -c19- | grep -c "^rw"
52484
$ find ~/.cache/ -type f -ls | cut -c19- | grep -cv "^rw"
2
$ find ~/.cache/ -type f -ls | cut -c19- | grep -v "^rw"
r--r--r--   1 user user      482 Apr 29 14:43 /home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/config.cue
r--r--r--   1 user user       94 Apr 29 14:43 /home/user/.cache/cue/github.com/cue-labs/examples/[email protected]/cue.mod/module.cue

[antong]

I think this is intentional. The cache fetch code specifically calls a function called makeDirsReadonly after extracting the zip. The Go module cache does the same. I don't know whether the main reasons for why Go does it applies to cue. The go command has a switch to leave directories writable (-modcacherw), and has a tool to clean the cache (go clean -modcache). Perhaps cue should as well?

A few notes: It is the permissions on the directory that prevents you from deleting a file. You can delete (unlink) a file that you do not have any permissions on, as long as you have write and execute permissions on the containing directory.

It is not really possible to not obey the umask. It is the OS that applies the umask by removing permissions masked by the umask. The umask is to make sure that programs don't create files with too lenient permissions. Disobeying the umask would be like for a program to chmod back permissions masked by the umask.

[myitcv]

cc @rogpeppe

• There appears to be a bug in that some files list in the original description are read-write when they should be read-only
• Otherwise I think things are working as intended here, modulo us ensuring we have good docs explaining that things in the cache are "at most" read only (because umask is taken into account) and #3131 linked below.

[myitcv]

Thanks for the detailed response here, @antong - I've raised https://github.com/cue-lang/cue/issues/3131 to capture the point about go clean -modcache.

[myitcv]

Assigning to v0.9.0-rc.1 for the bug here (which might well be related to https://github.com/cue-lang/cue/issues/3139), and because we should have good docs here ahead of modules going "live"

[mvdan]

As @antong explains, the umask behavior is correct. It declares the maximum permissions that newly created files should have, not the exact ones.

I'll send a patch shortly about documenting that the modules cache should in general not be modified by the user, and repurpose this issue to be about making all files in the module cache read-only, not just those extracted from zip archives.

[mvdan]

and repurpose this issue to be about making all files in the module cache read-only, not just those extracted from zip archives.

We will look into this as part of v0.10, as it is not a blocker for v0.9.

[rogpeppe]

I'll send a patch shortly about documenting that the modules cache should in general not be modified by the user, and repurpose this issue to be about making all files in the module cache read-only, not just those extracted from zip archives.

After some discussion, it seems that we probably don't need to do this:

• Go does not mark these zip files and mod files as read-only
• we can't make the directories read-only, because we need to be able to create lock files in them
• the main risk we're guarding against by making the files read-only is people editing them in their editors, which is much more of a danger for regular source files in extracted contents.

[seh]

This issue is closed, but per my comment in the "modules" channel of the "CUE" Slack workspace, it's not clear to me how I'm supposed to be able to clean up the CUE cache.

The CUE files that remain there—for which I lack any writing permission—prevent the rm -rf command from succeeding against any level of the directory tree that servers as the module cache.

[rogpeppe]

This issue is closed, but per my comment in the "modules" channel of the "CUE" Slack workspace, it's not clear to me how I'm supposed to be able to clean up the CUE cache.

The CUE files that remain there—for which I lack any writing permission—prevent the rm -rf command from succeeding against any level of the directory tree that servers as the module cache.

We should add a cue clean command similar to go clean (that's #3131), but in the meantime, I work around its lack by doing chmod -R +w $HOME/.cache/cue && rm -r $HOME/.cache/cue.

[seh]

I work around its lack by doing chmod -R +w $HOME/.cache/cue && rm -r $HOME/.cache/cue.

It's amazing that we have permission to give ourselves this permission, but yes, that works well enough to prove that it's possible. I'll be including that as a(n ideally temporary) workaround in my seh/rules_cue repository for the Bazel ruleset.

[myitcv]

@seh - I think you're after https://github.com/cue-lang/cue/issues/3131 as the answer for "I shouldn't have to change permissions like this"?

[seh]

Yes, that would help here, with a more explicit and deliberate approach to "remove all of the files and directories that CUE created for cached modules". I assume that I have permission to create the directory nominated by the "CUE_CACHE_DIR" environment variable, then I should have full permission to do anything I want to its contents.

Now, as precedent against that assertion, Bazel creates files and symbolic links in its "sandbox" directories that deny this permission, but those directory entries are an implementation detail that Bazel tries to hide from its users. It takes an explicit command-line flag (--sandbox_debug) to coax Bazel into leaving the directory tree in place for subsequent inspection. There is then the bazel clean command to remove these directories, among other things.