`utimensat` fails on `mountpoint-s3` files

[2022tgoel]

Description

I am trying to move a file into directory mounted on the host with mountpoint and getting the following error:

root@ce843bdb8afa:/# mv r.txt test-mount/
mv: preserving times for 'test-mount/r.txt': Operation not permitted

This operation succeeds fine with runc, and the issue seems to me to be that utimensat is returning with operation not permitted on the file that is mounted.

Steps to reproduce

Application code moving the file:

import modal
from modal import Image, Secret, CloudBucketMount

app = modal.App("copy-example")

cuda_image = (
    Image.debian_slim()
)


@app.function(
    image=cuda_image,
    volumes={
        "/mnt/output": CloudBucketMount(
            bucket_name="modal-s3mount-test-bucket",
            secret=Secret.from_name("aws-secret", required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"]),
            read_only=False
        ),
    }
)
def move():
    import shutil
    temp_output = "/r.txt"
    output_filename = "/mnt/output/output.txt"
    with open(temp_output, "wb") as f:
        shutil.move(temp_output, output_filename)

@app.local_entrypoint()
def main():
    move.remote()


if __name__ == "__main__":
    pass

runsc version

runsc version c5c74f4c90ac
spec: 1.1.0-rc.1

docker version (if using docker)

Docker version 27.3.1, build ce12230

uname

Linux ip-10-1-1-135.ec2.internal 6.11.0-0.el9ueknext.x86_64 #2 SMP PREEMPT_DYNAMIC Thu Oct 24 05:41:36 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

repo state (if built from source)

No response

runsc debug logs (if available)

D0121 18:24:27.493213       1 usertrap_amd64.go:210] [   5:   5] Found the pattern at ip 7eaebcb1a948:sysno 16
D0121 18:24:27.493226       1 usertrap_amd64.go:122] [   5:   5] Allocate a new trap: 0xc000620480 26
D0121 18:24:27.493235       1 usertrap_amd64.go:223] [   5:   5] Apply the binary patch addr 7eaebcb1a948 trap addr 60820 ([184 16 0 0 0 15 5] -> [255 36 37 32 8 6 0])
I0121 18:24:27.493247       1 strace.go:567] [   5:   5] mv E ioctl(0x4 /test-mount/r.txt, 0x40049409, 0x3)
I0121 18:24:27.493258       1 strace.go:605] [   5:   5] mv X ioctl(0x4 /test-mount/r.txt, 0x40049409, 0x3) = 0 (0x0) errno=25 (not a typewriter) (1.174µs)
D0121 18:24:27.493279       1 usertrap_amd64.go:210] [   5:   5] Found the pattern at ip 7eaebcb18d37:sysno 221
D0121 18:24:27.493284       1 usertrap_amd64.go:122] [   5:   5] Allocate a new trap: 0xc000620480 27
D0121 18:24:27.493289       1 usertrap_amd64.go:223] [   5:   5] Apply the binary patch addr 7eaebcb18d37 trap addr 60870 ([184 221 0 0 0 15 5] -> [255 36 37 112 8 6 0])
I0121 18:24:27.493353       1 strace.go:570] [   5:   5] mv E fadvise64(0x3 /r.txt, 0x0, 0x0, 0x2)
I0121 18:24:27.493363       1 strace.go:608] [   5:   5] mv X fadvise64(0x3 /r.txt, 0x0, 0x0, 0x2) = 0 (0x0) (556ns)
D0121 18:24:27.493435       1 usertrap_amd64.go:210] [   5:   5] Found the pattern at ip 7eaebcb1ea20:sysno 9
D0121 18:24:27.493443       1 usertrap_amd64.go:122] [   5:   5] Allocate a new trap: 0xc000620480 28
D0121 18:24:27.493449       1 usertrap_amd64.go:223] [   5:   5] Apply the binary patch addr 7eaebcb1ea20 trap addr 608c0 ([184 9 0 0 0 15 5] -> [255 36 37 192 8 6 0])
I0121 18:24:27.493498       1 strace.go:576] [   5:   5] mv E mmap(0x0, 0x22000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0xffffffffffffffff (bad FD), 0x0)
I0121 18:24:27.493540       1 strace.go:614] [   5:   5] mv X mmap(0x0, 0x22000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0xffffffffffffffff (bad FD), 0x0) = 139288957579264 (0x7eaebcd67000) (28.022µs)
I0121 18:24:27.493626       1 strace.go:567] [   5:   5] mv E read(0x3 /r.txt, 0x7eaebcd68000, 0x20000)
I0121 18:24:27.493647       1 strace.go:605] [   5:   5] mv X read(0x3 /r.txt, , 0x20000) = 0 (0x0) (11.717µs)
D0121 18:24:27.493664       1 usertrap_amd64.go:210] [   5:   5] Found the pattern at ip 7eaebcb19277:sysno 280
D0121 18:24:27.493672       1 usertrap_amd64.go:122] [   5:   5] Allocate a new trap: 0xc000620480 29
D0121 18:24:27.493678       1 usertrap_amd64.go:223] [   5:   5] Apply the binary patch addr 7eaebcb19277 trap addr 60910 ([184 24 1 0 0 15 5] -> [255 36 37 16 9 6 0])
I0121 18:24:27.493689       1 strace.go:570] [   5:   5] mv E utimensat(0x4 /test-mount/r.txt, <null>, 0x7ec711a98c30 {sec=1737483855 nsec=71940244}, 0x0)
I0121 18:24:27.504782       1 strace.go:608] [   5:   5] mv X utimensat(0x4 /test-mount/r.txt, <null>, 0x7ec711a98c30 {sec=1737483855 nsec=71940244}, 0x0) = 0 (0x0) errno=1 (operation not permitted) (11.080018ms)

[manninglucas]

I believe mountpoint-s3 is a FUSE, and gVisor's utimensat implementation calls vfs.SetStatAt to update timestamps. It could be an issue with our FUSE implementation.