lima-vm
chmod failed when `mountInotify` is enabled
Overview
After enabling the mountInotify feature, files on virtiofs faied to chmod under specific conditions.
I tried some debugging but could not figure out the cause.
Environment
- MacBook Pro(Apple M3)
- macOS 14.5(Sonoma)
$ limactl --version
limactl version 0.22.0
$ uname -v
Darwin Kernel Version 23.5.0: Wed May 1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031
Steps to reproduce
1. Create a template file
debug-inotify.yaml
vmType: "vz"
networks:
- vzNAT: true
mountType: "virtiofs"
mountInotify: true
images:
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img"
arch: "aarch64"
digest: "sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22"
mounts:
- location: "~"
writable: false
- location: "/tmp/lima"
writable: true
containerd:
system: false
user: false
2. Create a VM
$ limactl start --name=debug-inotify ./debug-inotify.yaml
$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ sudo apt -y install unzip
kuro@lima-debug-inotify:~$ exit
3. Download zip file(ex. terraform) and extract
$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ cd /tmp/lima/
kuro@lima-debug-inotify:/tmp/lima$ wget https://releases.hashicorp.com/terraform/1.9.0/terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ ls
terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ unzip terraform_1.9.0_darwin_arm64.zip
Archive: terraform_1.9.0_darwin_arm64.zip
inflating: LICENSE.txt
inflating: terraform fchmod (file attributes) error: Operation not permitted
(warning) cannot set modif./access times
Operation not permitted
kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro 4922 Jun 27 00:59 LICENSE.txt
-rw-r--r-- 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ chmod +x ./terraform
kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro 4922 Jun 27 00:59 LICENSE.txt
-rwxr-xr-x 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip
The error fchmod (file attributes) error: Operation not permitted occurred and the binary terraform does not have executable flag. However, after chmod +x manually, it have executable flag.
Debug
strace
kuro@lima-debug-inotify:/tmp/lima$ strace unzip terraform_1.9.0_darwin_arm64.zip
...[snip]
read(3, "\351\273\215\23\214\344\202\335\373y\302\366\326\300\241Q\216\356X\30\221\204\362\304\240\10\30\30\tK\360 "..., 8192) = 3818
write(4, "\2\270\374\t\365\343\362<\373:8\3335\360\306Q\376?\245j`coL\272\202\234\360\30r\374\247"..., 35024) = 35024
fchmod(4, 0100755) = -1 EPERM (Operation not permitted)
dup(2) = 5
fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(5, "fchmod (file attributes) error: "..., 56fchmod (file attributes) error: Operation not permitted
) = 56
close(5) = 0
...[snip]
fchmod(4, 0100755) = -1 EPERM (Operation not permitted) seems failed to chmod.
Customize guestagent
I implemented three types of function HandleInotify.
https://github.com/lima-vm/lima/blob/38c62c5e815fdc93e6e88543be1eb9ef7699eaf5/pkg/guestagent/guestagent_linux.go#L337-L346
a. Remove os.Ctimes()
This causes the same issue.
func (a *agent) HandleInotify(event *api.Inotify) {
location := event.MountPath
if _, err := os.Stat(location); err == nil {
logrus.Infof("inotify handle. Event: %s", event)
}
}
b. Remove everything
This does NOT cause problems.
func (a *agent) HandleInotify(event *api.Inotify) {
logrus.Infof("inotify handle. Event: %s", event)
}
c. Insert delay before os.Stat()
This does NOT cause problems. (100ms is bad, but 500ms or later is ok)
func (a *agent) HandleInotify(event *api.Inotify) {
location := event.MountPath
go func(){
time.Sleep(500 * time.Millisecond)
if _, err := os.Stat(location); err == nil {
logrus.Infof("inotify handle. Event: %s", event)
}
}()
}
The reason for adding os.Ctimes() is to simulate a modification. Only then guest can understand that a change happened for that file
@balajiv113 I understand your point. I was investigating to isolate the cause of this problem.
As a result, I have noticed that chmod fails regardless of whether or not os.Ctimes() is done.
As a result, I have noticed that chmod fails regardless of whether or not os.Ctimes() is done.
Exactly, this is what i was expecting. Our code related to inotify should not have any impact in unzip flow. This will be mostly a change needed from unzip itself or the mount type (virtiofs / 9p) doesn't support it.
FYI: I tried to decompress a tar.gz file with permission and this was working fine
I reproduced with tar.gz. Perhaps it has something to do with file size. Small file sizes(e.g. 1KB) did not cause errors.
kuro@lima-debug-inotify:/tmp/lima$ dd if=/dev/urandom of=test.bin bs=50M count=1
1+0 records in
1+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.110755 s, 473 MB/s
kuro@lima-debug-inotify:/tmp/lima$ chmod +x test.bin
kuro@lima-debug-inotify:/tmp/lima$ tar zcvf test.tar.gz test.bin
test.bin
kuro@lima-debug-inotify:/tmp/lima$ tar zxvf ./test.tar.gz
test.bin
tar: test.bin: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
strace result:
kuro@lima-debug-inotify:/tmp/lima$ strace tar zxvf ./test.tar.gz
...[snip]
write(4, "\3519\34a\244\2300\4\304\236GO\202\241\256\vK\361\213\224v\250\355x\341\237\204Se\373-\355"..., 512) = 512
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1719503700, tv_nsec=0} /* 2024-06-28T00:55:00+0900 */], 0) = -1 EPERM (Operation not permitted)
write(2, "tar: ", 5tar: ) = 5
write(2, "test.bin: Cannot utime", 22test.bin: Cannot utime) = 22
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
tar throws for utime only. But the file should be unzipped with proper permission