[Error] On Silverblue, Ubuntu container fails to apt-install due to /etc/localtime mount

[javster101]

Describe the bug On Silverblue 36, when installing packages in an Ubuntu 22.10 container, apt-get eventually fails due to a distrobox preinvoke script. This also happens when trying to unmount manually. I've already restarted the container, restarted Silverblue, and recreated the container a few times (always happens after some time using a new container). Running lsof /etc/localtime in Silverblue and in the Ubuntu container shows nothing (same as running it in an Arch container). If it helps, I'm running kitty+zsh as my terminal.

Logs

❯ sudo apt install fuse
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bridge-utils dns-root-data dnsmasq-base iproute2 libatm1 libbpf0 libcap2-bin libpam-cap netcat-openbsd
  ubuntu-fan
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libfuse2
The following NEW packages will be installed:
  fuse libfuse2
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/117 kB of archives.
After this operation, 446 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
/etc/localtime is a mountpoint
umount: /etc/localtime: target is busy.
E: Problem executing scripts DPkg::Pre-Invoke 'if mountpoint /etc/localtime; then umount /etc/localtime; fi'
E: Sub-process returned an error code

Desktop (please complete the following information):

• Are you using podman or docker? Podman
• Which version or podman or docker? 4.2.0
• Which version of distrobox? 1.4.0

Additional context It doesn't happen immediately, I'm unsure at what point during using the container this starts.

Seems to be caused by this commit: https://github.com/89luca89/distrobox/commit/befabd34adb61019649f822e81e2e7d559bb2948

[javster101]

Commenting out the first two lines of the generated /etc/apt/apt.conf.d/00_distrobox seems to fix it.

#DPkg::Pre-Invoke {"if mountpoint /etc/localtime; then umount /etc/localtime; fi";};
#DPkg::Post-Invoke {"if [ -e /run/host//etc/localtime ]; then mount --rbind /run/host//etc/localtime /etc/localtime; fi";};
DPkg::Pre-Invoke {"if mountpoint /run/systemd/journal; then umount /run/systemd/journal; fi";};
DPkg::Post-Invoke {"if [ -e /run/host//run/systemd/journal ]; then mount --rbind /run/host//run/systemd/journal /run/systemd/journal; fi";};
DPkg::Pre-Invoke {"if mountpoint /run/systemd/resolve; then umount /run/systemd/resolve; fi";};
...

[luc14n0]

I can confirm the behavior in both my Ubuntu 22.04 and Debian Unstable containers.

❯ lsb-release -dr
Description:    openSUSE Tumbleweed
Release:        20220915

❯ rpm -q podman distrobox
podman-4.2.0-1.2.x86_64
distrobox-1.4.0-1.1.noarch

[89luca89]

Thanks for reporting, will investigate :+1:

I'll probably need some help from you, because sadly I cannot reproduce this on 22.04 nor 22.10:

---

Generally speaking /etc/localtime is a mountpoint to host's localtime, so that timezones are in sync and in the pre-install hook, I try to umount it to fix compatibility with some packages like tzdata

[luc14n0]

In my case, lsof | grep /etc/localtime showed me that gitstatus was the one keeping the mount point busy. This is a tool used by Rowan's Powerlevel10k Zsh prompt theme, I'm guessing it's related with the clock in the right prompt:

In a test I did, if I disable Powerlevel10k's prompt apparently I don't get the "target is busy" error anymore. However, the post-install hook DPkg::Post-Invoke {"if [ -e /run/host//etc/localtime ]; then mount --rbind /run/host//etc/localtime /etc/localtime; fi";}; doesn't mount /etc/localtime back.

[javster101]

That would make sense, I'm also using Powerlevel10k.

[89luca89]

In my case, lsof | grep /etc/localtime showed me that gitstatus was the one keeping the mount point busy. This is a tool used by Rowan's Powerlevel10k Zsh prompt theme, I'm guessing its related with the clock in the right prompt:

In a test I did, if I disable Powerlevel10k's prompt apparently I don't get the "target is busy" error anymore. However, the post-install hook DPkg::Post-Invoke {"if [ -e /run/host//etc/localtime ]; then mount --rbind /run/host//etc/localtime /etc/localtime; fi";}; doesn't mount /etc/localtime back.

This is probably because you previously umounted it to test Try stopping, and starting again the container, without Powerlevel10k, and see if the apt command works

If it's an incompatibility with Powerlevel10k keeping the file open, I cannot do much here :shrug:

[luc14n0]

This is probably because you previously umounted it to test Try stopping, and starting again the container, without Powerlevel10k, and see if the apt command works

Actually I didn't. I should've been more explicit there.

The steps I did were:

1. Disable Powerlevel10k.
2. Restart the container.
3. Check the mount point with mount | grep /etc/localtime (it was there of course).
4. Install something with # apt install (it worked without any errors).
5. Check the mount point again with mount | grep /etc/localtime (it wasn't back there).
6. Try to manually mount it back, but no dice.

---

I then did a bit of a testing:

I'm not sure if other distros also use a symlink for /etc/localtime, but the symlink gets stale inside the containers, probably because the host's /usr/share is not exposed to the containers.

P.S. I had a Gnome OS VM here, and it also uses a symlink there, so it seems to be some what common-ish.

If it's an incompatibility with Powerlevel10k keeping the file open, I cannot do much here shrug

Yeah, I agree. gitstatus keeps about a dozen instances of opened /etc/localtime, maybe because I'm using Tmux. Anyway, I'm gonna mule over this issue and see if I can come up with any work around/fix myself, be it on the Powerlevel10k side or Distrobox's. If by any chance I find an elegant solution on our side I can open a MR at some point. For now I'm gonna add a logic to my Powerlevel10k config file so the clock (the probable guy asking for /etc/localtime) is disabled for Distrobox containers.

[luc14n0]

I then did a bit of a testing:

I'm not sure if other distros also use a symlink for /etc/localtime, but the symlink gets stale inside the containers, probably because the host's /usr/share is not exposed to the containers.

P.S. I had a Gnome OS VM here, and it also uses a symlink there, so it seems to be some what common-ish.

I had a breakthrough yesterday, it appears that because I was having trouble installing any package, and at some point I disabled the pre/post hooks to (u)mount /etc/localtime to allow their installation, only tzdata (since tzdata is related directly with /etc/localtime of course would still fail to install).

So now that I installed tzdata I don't have a broken /run/host/etc/localtime -> /usr/share/zoneinfo/America/Sao_Paulo symlink anymore and the post DPkg::Post-Invoke {"if [ -e /run/host//etc/localtime ]; then mount --rbind /run/host//etc/localtime /etc/localtime; fi";} hook is now functional.

For me the issue here got sorted, I just have to figure out how to make Powerlevel10k to coexist with Distrobox (only disabling the clock is not enough to keep gitstatus's hands out of /etc/localtime). On my account, this issue can get closed.

[javster101]

I guess the question is, then, should applications that keep /etc/localtime open just be forced to close? There's gotta be other applications that just constantly check the time. Ideally a message should be written to the terminal at least if this issue is detected.

[89luca89]

should applications that keep /etc/localtime open just be forced to close?

That would be a problem if it's your prompt no? :smile:

Ideally a message should be written to the terminal at least if this issue is detected.

Better yes, will probably implement something on these lines

[luc14n0]

Yeah, forcefully closing apps that use /etc/localtime or any other thing in any mount point, thus keeping them busy, doesn't sound the way to go.

A message at least stating what the problem is, in our case here the mount point was being kept busy so it couldn't be unmounted, would be useful to some people but still a bit cryptic. If a new dependency was to be added, let's say on lsof, a message more specific could be generated though.

P.S.

Actually we already get a:

/etc/localtime is a mountpoint
umount: /etc/localtime: target is busy.
E: Problem executing scripts DPkg::Pre-Invoke 'if mountpoint /etc/localtime; then umount /etc/localtime; fi'
E: Sub-process returned an error code

So we get a WHAT is the issue but no WHY.

[javster101]

Yeah, I meant the force-close thing more to just make a point. If it could run lsof and tell you which application was using the file that is being unmounted that would be helpful. The ideal thing would be to only try to unmount if installing packages that require an unmount, but I imagine that hardcoding a list of packages would be a bit of a pain.

[luc14n0]

Hey @javster101, I've came up with a workaround until I can find out why gitstatus keep many instances of /etc/localtime opened at all times, or a more elegant solution. I won't put it here to lessen extra noise, but you can find 2 gists I made for it in my namespace (I won't link them either).

[89luca89]

@javster101 @luc14n0

The lsof solution could work, it is one of the basic packages already installed by the distrobox-init so it should be ok

In the future I'll add a message as suggested Or if anyone wants to give it a shot, is always welcome :smile:

[luc14n0]

@javster101 @luc14n0

The lsof solution could work, it is one of the basic packages already installed by the distrobox-init so it should be ok

In the future I'll add a message as suggested Or if anyone wants to give it a shot, is always welcome :smile:

Ah, it's already there, then I'm going to start some experiments.

[henriqueffc]

I then did a bit of a testing: I'm not sure if other distros also use a symlink for /etc/localtime, but the symlink gets stale inside the containers, probably because the host's /usr/share is not exposed to the containers. P.S. I had a Gnome OS VM here, and it also uses a symlink there, so it seems to be some what common-ish.

I had a breakthrough yesterday, it appears that because I was having trouble installing any package, and at some point I disabled the pre/post hooks to (u)mount /etc/localtime to allow their installation, only tzdata (since tzdata is related directly with /etc/localtime of course would still fail to install).

So now that I installed tzdata I don't have a broken /run/host/etc/localtime -> /usr/share/zoneinfo/America/Sao_Paulo symlink anymore and the post DPkg::Post-Invoke {"if [ -e /run/host//etc/localtime ]; then mount --rbind /run/host//etc/localtime /etc/localtime; fi";} hook is now functional.

For me the issue here got sorted, I just have to figure out how to make Powerlevel10k to coexist with Distrobox (only disabling the clock is not enough to keep gitstatus's hands out of /etc/localtime). On my account, this issue can get closed.

It worked for me. Thanks.

[kramfs]

I'm also experiencing this issue. On a fresh Ubuntu container, apt installation works but once you install the git package, the issue starts every time apt install/remove is used. And yes, I'm also using zsh with Powerlevel10k theme.

E: Problem executing scripts DPkg::Pre-Invoke 'if findmnt /etc/localtime >/dev/null; then umount /etc/localtime; fi'
E: Sub-process returned an error code

I've commented /etc/apt/apt.conf.d/00_distrobox on the container, but this needs to be set on every container that is created, not an elegant solution but a crude workaround.

[black-desk]

@kramfs For me, I just use lsof to check who open /etc/localtime, and kill all the process.

For p10k, kill gitstatus will make zsh print some error as broken pipe or something else.

After run apt, exec zsh will bring gitstatus back as good as before. I think that is a good enought work around.

[89luca89]

Well the point is:

• if I don't umount /etc/localtime during an apt transaction, it might fail if it tries to touch that file (eg: tz-data)
• if it's busy because a process is keeping it open, I cannot just automatically kill it, it could create very bad situations

What to do? :thinking:

[black-desk]

• if I don't umount /etc/localtime during an apt transaction, it might fail if it tries to touch that file (eg: tz-data)

Is there any issue link?

[89luca89]

Glad to say I've found a fix for this, I've tested with Powerline10k and o-m-zsh and all works well now :+1:

[black-desk]

Glad to say I've found a fix for this, I've tested with Powerline10k and o-m-zsh and all works well now 👍

[89luca89]

@black-desk did you stop and restart the container after upgrade distrobox on the host? EDIT:

I've tried on various distributions host , the same ubuntu+p9k+omzsh setup and it's now working as intended

[black-desk]

@black-desk did you stop and restart the container after upgrade distrobox on the host?

I even delete then recreate the container.

But the way, I am using WSL2.

[89luca89]

@black-desk did you stop and restart the container after upgrade distrobox on the host?

I even delete then recreate the container.

But the way, I am using WSL2.

That is not really a supported platform :eyes: I don't have a windows machine to test tbh so I don't know if it's any different from another native host

[black-desk]

That is not really a supported platform 👀 I don't have a windows machine to test tbh so I don't know if it's any different from another native host

I will have a try on a real machine later.

Saidly, distrobox and podman's toolbox both not working out of box in WSL2. :(

[89luca89]

This also is a separate issue from this, that one is tzdata upgrade problem, this issue is related to a compatibility with p9k prompts.

Anyway on a real machine, tzdata upgrade works: