try icon indicating copy to clipboard operation
try copied to clipboard
verified publisher icon binpash

Could not delete created temporary path

Open zdyxry opened this issue 2 years ago • 10 comments

OS info:

➜  try git:(main) ✗ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Minimal Reproducible Script：

#!/usr/bin/bash

cur=$(pwd)

sudo rm -rf /tmp/tmp.*
$cur/try -y touch $HOME/testfile
ls -l /tmp/testfile
ls -l /tmp/tmp.*
rm -rf /tmp/tmp.*
$cur/try -y rm $HOME/testfile
ls -l /tmp/testfile
ls -l /tmp/tmp.*
rm -rf /tmp/tmp.*

Output logs:

➜  try git:(main) ✗ sh a.sh
Warning: Failed mounting /boot as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /snap as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /var as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /snap/bare/5 as an overlay, see /tmp/tmp.iWAZWwEcHw
...
Warning: Failed mounting /snap/firefox/2800 as an overlay, see /tmp/tmp.iWAZWwEcHw
ls: cannot access '/tmp/testfile': No such file or directory
-rw------- 1 yiran yiran 2668  6月 25 17:28 /tmp/tmp.iWAZWwEcHw
-rwx--x--x 1 yiran yiran  120  6月 25 17:28 /tmp/tmp.pU1iJHkqNZ
-rwx--x--x 1 yiran yiran 1824  6月 25 17:28 /tmp/tmp.ZJln7EqgWV

/tmp/tmp.a5YeVGkibC:
total 12
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 temproot
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 upperdir
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 workdir
Warning: Failed mounting /boot as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /var as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/bare/5 as an overlay, see /tmp/tmp.1amwhWmW0M
...
Warning: Failed mounting /var/snap/firefox/common/host-hunspell as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/snapd/19361 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /boot/efi as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core18/2785 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/slack/79 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/gnome-42-2204/111 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/snapd/19457 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core20/1950 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core22/766 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/firefox/2800 as an overlay, see /tmp/tmp.1amwhWmW0M
ls: cannot access '/tmp/testfile': No such file or directory
-rw------- 1 yiran yiran 2668  6月 25 17:28 /tmp/tmp.1amwhWmW0M
-rwx--x--x 1 yiran yiran  117  6月 25 17:28 /tmp/tmp.iNGI2499F3
-rwx--x--x 1 yiran yiran 1824  6月 25 17:28 /tmp/tmp.LjL0GhNZOQ

/tmp/tmp.j8aZKEw1zb:
total 12
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 temproot
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 upperdir
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 workdir
rm: cannot remove '/tmp/tmp.j8aZKEw1zb/workdir/home/work': Permission denied

The file path operated by the try command will remain, and then cannot be deleted by the current user.

zdyxry avatar Jun 25 '23 09:06 zdyxry

A few things will help us debug this:

  • Can you tell us how your /tmp is mounted?
  • Can you please show the permissions of the files in workdir at each stage?

Thanks!

mgree avatar Jun 25 '23 13:06 mgree

/tmp dir info:

➜  try git:(main) ✗ mount |grep tmp
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16393976k,nr_inodes=4098494,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3287196k,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /run/user/125 type tmpfs (rw,nosuid,nodev,relatime,size=3287192k,nr_inodes=821798,mode=700,uid=125,gid=130,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3287192k,nr_inodes=821798,mode=700,uid=1000,gid=1000,inode64)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3287196k,mode=755,inode64)
tmpfs on /run/qemu type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
➜  try git:(main) ✗ ll / |grep tmp
drwxrwxrwt  25 root root  60K  6月 26 08:10 tmp

Exec ./try touch $HOME/testfile prompt Commit these changes? [y/N] tmp dir permissions:

➜  /tmp ll -trh
drwxrwxr-x 8 yiran yiran 4.0K  6月 25 17:42 try
drwx------ 5 yiran yiran 4.0K  6月 26 08:12 tmp.N2h9j5rPDT
-rwx--x--x 1 yiran yiran 1.8K  6月 26 08:12 tmp.N11kDiHPxu
-rwx--x--x 1 yiran yiran  121  6月 26 08:12 tmp.bFLaUHf8mR
-rw------- 1 yiran yiran 2.7K  6月 26 08:12 tmp.812oSzBLj0
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT
total 12K
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 workdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 upperdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 temproot
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir
total 108K
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 dev
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 proc
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 run
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 sys
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 bin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 cdrom
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 boot
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 etc
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 home
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib64
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lost+found
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 libx32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 media
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 mydir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 mnt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 nix
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 opt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 root
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 snap
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 sbin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 srv
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 tmp
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 usr
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 yiran
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 var
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:12 work

Exec ./try rm $HOME/testfileprompt Commit these changes? [y/N] tmp dir permissions:

➜  /tmp ll -trh tmp.pmesVRCcFi
total 12K
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 workdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 upperdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 temproot
➜  /tmp ll -trh tmp.pmesVRCcFi/workdir
total 108K
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 dev
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 run
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 proc
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 sys
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 boot
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 bin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 cdrom
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 etc
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 home
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib64
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lost+found
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 libx32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 media
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 mydir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 mnt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 nix
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 opt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 root
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 snap
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 sbin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 srv
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 tmp
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 var
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 usr
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 yiran
➜  /tmp ll -trh tmp.pmesVRCcFi/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:15 work

When I commit these changes and try to delete the temporary directory:

➜  /tmp rm -rf tmp.pmesVRCcFi
rm: cannot remove 'tmp.pmesVRCcFi/workdir/home/work': Permission denied
➜  /tmp tree tmp.pmesVRCcFi
tmp.pmesVRCcFi
└── workdir
    └── home
        └── work  [error opening dir]

3 directories, 0 files

zdyxry avatar Jun 26 '23 00:06 zdyxry 

in fs
drwxrwxrwt  25 root root  60K  6月 26 08:10 tmp

in workdir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 tmp

Could this be also because of tmp is losing 777t in overlay #6 ?

ezrizhu avatar Jun 26 '23 00:06 ezrizhu

I also tried it with AlmaLinux 9 and it didn't reproduce the problem.

Do you have any debugging experience to share?

zdyxry avatar Jun 26 '23 00:06 zdyxry

So it seems that /tmp is an ordinary directory.

➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:12 work

indicates that overlayfs's in-flight work is getting no perms at all, which seems odd. (Though I've never checked to see what overlayfs does with it, and I believe it's always empty by the time I've committed.)

These logs don't quite cover all of our bases---what's $HOME set to, and is it in the same mount as /tmp? What's the FS type of /?

mgree avatar Jun 26 '23 17:06 mgree

Hi , $HOME is an ordinary directory, / filesystem is ext4 .

➜  try git:(main) ✗ echo $HOME
/home/yiran
➜  try git:(main) ✗ mount |grep -i home
➜  try git:(main) ✗ ll / |grep home
drwxr-xr-x   3 root root 4.0K  8月 12  2021 home
➜  try git:(main) ✗ ll /home
total 4.0K
drwxr-xr-x 61 yiran yiran 4.0K  6月 27 08:43 yiran
➜  try git:(main) ✗ findmnt| head -n 2
TARGET                                                                                             SOURCE                         FSTYPE              OPTIONS
/                                                                                                  /dev/sda2                      ext4                rw,relatime,errors=remount-ro
➜  try git:(main) ✗ lsblk |grep -iv snap # exclude snap mount points
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   200G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0 199.5G  0 part /var/snap/firefox/common/host-hunspell
                                 /
sr0     11:0    1  1024M  0 rom

This doesn't seem to be related to $HOME, changing $HOME to /tmp/ has the same result.

zdyxry avatar Jun 27 '23 00:06 zdyxry

Just wanted to be sure that $HOME didn't matter. The ext4 rules out underlying fs weirdness.

What kernel version are you running?

mgree avatar Jun 27 '23 11:06 mgree

kernel is the default version provided by Ubuntu:

kernel version and /proc/cmdline :

➜  try git:(main) ✗ uname -a
Linux yiran-virtual-machine 5.19.0-43-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
➜  try git:(main) ✗ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.19.0-43-generic root=UUID=77e940a3-e385-4f7f-ba35-b2506cf25be9 ro quiet splash vt.handoff=7

zdyxry avatar Jun 28 '23 00:06 zdyxry

Huh. I'm seeing something similar in the (failing) test case of #66, which tells me that cleaning these up is not trivial and/or something is weird in overlayfs's workdir.

mgree avatar Jun 28 '23 00:06 mgree

Okay: work for this will proceed in #93, so I'm marking this as a duplicate.

mgree avatar Jun 29 '23 20:06 mgree