"Operation not permitted" when trying to open file for write

[stsp]

This may be related to #232 but I prepared a ready-to-use test-case, it is attached: fuse_test.AppImage.gz

After you gunzip it, you'll get the AppImage file called fuse_test.AppImage. All it does is mounts the read-only lower dir and the writable upper dir into the mount-point. Then it does test -r and test -w to check if the file is actually readable and writable. Unfortunately its not writable, in which case it prints the failure message and also prints mount | grep fuse that allows you to inspect the mount params. In fact, its not like the entire dir is not writable - it IS writable. You can create new files there. But the files that come from the lower dir, are themselves not writable for some unclear reason.

I also added the --debug option for you. With that option fuse-overlayfs is started with -d and you can see the logging. I also added the --keep option which just mounts the dirs, prints the mount point and exits without doing any checks and without unmounting. This may be needed if you want to work with the mount-point by hands.

I hope this test-case will help to narrow down the problem.

[stsp]

ovl_tst.tar.gz Here's the completely trivial test-case that doesn't even involve AppImage. Just a trivial script. It mounts the archive with archivemount and then adds a writable overlay. The very interesting thing is that the bug can be reproduced only with SOME archives. The one that is attached, triggers the bug. But if I try to create the similar archive by hands, then bug doesn't trigger... So the problem is probably much deeper than the one could expect.

[stsp]

Found the problem: fuse-overlayfs doesn't make the files writable if in the lowerdir they belong to another user.

[stsp]

In fact, the ownership is propagated from lowerdir to the mountpoint. Maybe fuse-overlayfs should set the ownership to the mount owner, and ignore the ownership in the lowerdir?

[probonopd]

I think what should be done is to do copy-on-write under the user running fuse-overlayfs while not caring about the user of the file in the lowerdir?

[stsp]

I think what should be done is to do copy-on-write under the user running fuse-overlayfs while not caring about the user of the file in the lowerdir?

I bet the write request doesn't even come to fuse-overlayfs in that case. Kernel's FUSE module has paranoid permission checks, and likely denies the request if the file has another owner. I think the only work-around is for fuse-overlayfs to change the owner explicitly.

[stsp]

Got things working with this hack:

diff --git a/main.c b/main.c
index b5753db..58836c8 100644
--- a/main.c
+++ b/main.c
@@ -971,8 +971,8 @@ rpl_stat (fuse_req_t req, struct ovl_node *node, int fd, con
st char *path, struc
   if (ret < 0)
     return ret;
 
-  st->st_uid = find_mapping (st->st_uid, data, true, true);
-  st->st_gid = find_mapping (st->st_gid, data, true, false);
+  st->st_uid = find_mapping (getuid(), data, true, true);
+  st->st_gid = find_mapping (getgid(), data, true, false);
 
   st->st_ino = node->tmp_ino;
   st->st_dev = node->tmp_dev;
@@ -3186,12 +3186,14 @@ copyup (struct ovl_data *lo, struct ovl_node *node)
   if (dfd < 0)
     goto exit;
 
+#if 0
   if (st.st_uid != lo->uid || st.st_gid != lo->gid || get_upper_layer (lo)->stat_override_mode != STAT_OVERRIDE_NONE)
     {
       ret = do_fchown (lo, dfd, st.st_uid, st.st_gid, mode);
       if (ret < 0)
         goto exit;
     }
+#endif
 
   buf = malloc (buf_size);
   if (buf == NULL)

[probonopd]

Would you be able to send a pull request?

[stsp]

This patch is definitely wrong, it can't be sent as a PR. I'll simply add the hacked-up fuse-overlayfs to my AppImage, and things are good.

[stsp]

Its not like I don't care about an upstream version, but fuse-overlayfs has quite a lot of uid mapping and overriding modes which I am not aware about. Maybe enabling some override mode already achieves the same thing, I've no idea. So I'll just take a short-cut and package the hacked-up version.

[probonopd]

It would be good if this could be fixed properly by the maintainer of this repo, so that it would be fixed for everyone. Thanks!

[boegel]

I'm running into similar problems, so I'm wondering if the proposed fix has been considered by the fuse-overlayfs maintainers.

@giuseppe Can you share your thoughts on this?

[giuseppe]

well. the patch is not correct, it ignores the file ownership.

In what environment is fuse-overlayfs running? Is it inside a user namespace with multiple IDs? The expectation is that it runs with CAP_DAC_OVERRIDE so it can access every ID mapped in the user namespace. If that is not the case, maybe we can add a new option to squash all the IDs to the same value

[stsp]

well. the patch is not correct

Yes.

In what environment is fuse-overlayfs running? Is it inside a user namespace with multiple IDs?

No, its in a main namespace. I provided you with a trivial test-case here: https://github.com/containers/fuse-overlayfs/issues/428#issuecomment-2268525669

[giuseppe]

I'd be fine with a patch that checks whether the current process has CAP_DAC_OVERRIDE and if it doesn't, then skip completely the chown, something like:

static int
do_fchown (struct ovl_data *lo, int fd, uid_t uid, gid_t gid, mode_t mode)
{
  int ret;

  if (! lo->has_cap_dac_override)  // TODO: calculate has_cap_dac_override
    return 0;

  if (lo->xattr_permissions)
    ret = write_permission_xattr (lo, fd, NULL, uid, gid, mode);
  else
    ret = fchown (fd, uid, gid);
  return (lo->squash_to_root || lo->squash_to_uid != -1 || lo->squash_to_gid != -1) ? 0 : ret;
}

would that be enough for your use-case?

[stsp]

Hmm, trying out my aforementioned test-case, and it seems to regress a lot since!

$ fuse-overlayfs -o lowerdir=lower -o upperdir=ov -o workdir=. rw
fusermount3: mount failed: Permission denied
fuse-overlayfs: cannot mount: Operation not permitted
$ ls -l `which fusermount3`
-rwsr-xr-x 1 root root 39376 сен 21  2024 /usr/bin/fusermount3

Wtf? fusermount3 is still suid-root, but "Permission denied"??? Ok, try sudo:

$ sudo fuse-overlayfs -o lowerdir=lower -o upperdir=ov -o workdir=. rw
unknown argument ignored: lazytime
cannot load store (null) at lower
fuse-overlayfs: cannot read lower dirs: Permission denied

What? "unknown argument ignored: lazytime"? Really? Well, I don't think I can test your patch. :(

Ubuntu-25.04