[Bug] Created user directories is owned by root, and not the user

[a12l]

My $HOME is rolled back to a blank ZFS snapshot at every startup. The dotfiles that I want to keep between reboots is listed in Impermanence's directory and file lists; and the other files that I want to keep is located inside ZFS datasets that gets mounted in $HOME.

My problem using the new way to list users' directories and files is that the parent directories is created and owned by root. This cause a lot of problems. For example, if I have this in my NixOS configuration

environment.persistence."/persistent".users.a12l.directories = [ ".local/share/fonts" ];

then I get this when I log in

$ ls -la|rg .local
drwxr-xr-x  4 root root     4 Feb  1 21:33 .local

$ ls -lR .local
.local:
total 33
drwxr-xr-x 3 root root 3 Feb  1 21:33 share

.local/share:
total 17
drwxr-xr-x 3 a12l users 9 Jan 18 15:39 fonts

.local/share/fonts:
total 852
-rw-r--r-- 1 a12l users  44732 Jan 18 15:31 all-the-icons.ttf
-rw-r--r-- 1 a12l users 489672 Jan 18 15:31 file-icons.ttf
-rw-r--r-- 1 a12l users 152796 Jan 18 15:31 fontawesome.ttf
-rw-r--r-- 1 a12l users 128180 Jan 18 15:31 material-design-icons.ttf
-rw-r--r-- 1 a12l users  52544 Jan 18 15:31 octicons.ttf
drwxr-xr-x 2 a12l users     18 Jan 30 03:37 p
-rw-r--r-- 1 a12l users  99564 Jan 18 15:31 weathericons.ttf

.local/share/fonts/p:
total 19744
-rw-r--r-- 1 a12l users 2379348 Jan 30 03:26 PragmataProB_0829.ttf
-rw-r--r-- 1 a12l users 2404140 Jan 30 03:26 PragmataProB_liga_0829.ttf
-rw-r--r-- 1 a12l users 2262112 Jan 30 03:26 PragmataProI_0829.ttf
-rw-r--r-- 1 a12l users 2287048 Jan 30 03:26 PragmataProI_liga_0829.ttf
-rw-r--r-- 1 a12l users 1326160 Jan 30 03:26 PragmataPro_Mono_B_0829.ttf
-rw-r--r-- 1 a12l users 1351116 Jan 30 03:26 PragmataPro_Mono_B_liga_0829.ttf
-rw-r--r-- 1 a12l users 1212296 Jan 30 03:26 PragmataPro_Mono_I_0829.ttf
-rw-r--r-- 1 a12l users 1239864 Jan 30 03:26 PragmataPro_Mono_I_liga_0829.ttf
-rw-r--r-- 1 a12l users 1950440 Jan 30 03:26 PragmataPro_Mono_R_0829.ttf
-rw-r--r-- 1 a12l users 1974976 Jan 30 03:26 PragmataPro_Mono_R_liga_0829.ttf
-rw-r--r-- 1 a12l users 1146460 Jan 30 03:26 PragmataPro_Mono_Z_0829.ttf
-rw-r--r-- 1 a12l users 1170496 Jan 30 03:26 PragmataPro_Mono_Z_liga_0829.ttf
-rw-r--r-- 1 a12l users 3182624 Jan 30 03:26 PragmataProR_0829.ttf
-rw-r--r-- 1 a12l users 3207080 Jan 30 03:26 PragmataProR_liga_0829.ttf
-rw-r--r-- 1 a12l users 2184928 Jan 30 03:26 PragmataProZ_0829.ttf
-rw-r--r-- 1 a12l users 2209928 Jan 30 03:26 PragmataProZ_liga_0829.ttf

Note that ~/.local and ~/.local/share is owned by root:root, while ~/.local/share/fonts and below is owned by a12l:users.

I expected that all directories that is automatically created by Impermanence should be owned by the user with the username listed in environment.persistence."/persistent".users.<user>, I.e. a12l in my case.

[etu]

I have noticed this as well, but one case that is extra interesting as well as the root owned one is this line: https://github.com/etu/nixconfig/blob/main/hosts/eliaxe-A100514-NR/persistence.nix#L60

I've expect ~/VirtualBox VMs to be created and mounted.

It is created in the tmpfs space and mounted correctly, so far so good.

But I also got ~/VirtualBox/VMs created and that entire file-tree is owned by root:root.

But that's a separate issue! But what I really did was to confirm your issue as well.

[talyz]

Whoops, this was because of a stupid last-minute "optimization" I did. Sorry about that. Should now be fixed in master.

@etu I'm seeing that issue as well. Everything seems to be working correctly, except for the directory creation, so there's probably something strange happening in create-directories.bash. I'll have to investigate it further, though.

[talyz]

Oh yeah, it uses space as a separator, so that's what's why. I'll fix it.

[a12l]

@talyz I think that this problem disappeared after you pushed your fix [1] when I still had a basic configuration. But now when my configuration [2] is based on digga [3] (a configuration framework building upon flake-utils-plus [4]), the problem still occurs. Note that I'm not at all sure if the problem disappeared with your fix while still using the "basic" configuration, but I think so. Sadly I don't have my old configuration files.

Addendum: I'm using the latest commit of Impermanence, as you can see in my lock file [5].

[1] https://github.com/nix-community/impermanence/commit/65caf299a582ef7cd14b586e8ca0ffe42a363613 [2] https://github.com/a12l/nixosConfig [3] https://github.com/divnix/digga [4] https://github.com/gytis-ivaskevicius/flake-utils-plus [5] https://github.com/a12l/nixosConfig/blob/trunk/flake.lock#L239

[talyz]

Note that it won't change the permission of any files that already exist in persistent storage. Can you provide an example where you add a new directory to the list and it ends up with incorrect permissions? Also look in the activate script for your current generation and search for the createPersistentStorageDirs snippet. Are the correct permissions provided to the create-directories script for each directory?

[a12l]

The problem is that ~/.config; ~/.local; and ~/.local/share is owned by root:root, and not a12l:users. The files that I've listed for persistent storage is located inside ~/.config and in ~/.local/share, but I haven't listed ~/.config nor ~/.local/share for persistent storage.

You can see below that the ~/.config; ~/.local; and ~/.local/share directories located inside the persistent storage has correct ownership. (I've removed files and directories that isn't handled by Impermanence for clarity)

$ ls -la /persistent/home/a12l/
total 175
drwxr-xr-x 10 a12l users   13 Feb  6 12:55 .
drwxr-xr-x  3 root root     3 Dec 16 10:49 ..
drwxr-xr-x 12 a12l users   14 Feb  6 12:53 .config
drwxr-xr-x  4 a12l users    4 Jan 25 22:59 .local
drwxr-xr-x  2 a12l users    2 Dec 23 00:35 .minisign
drwxr-xr-x  5 a12l users    5 Jan 16 20:10 .mozilla
drwxr-xr-x  2 a12l users    2 Dec 23 02:17 .scribus
drwxr-xr-x  2 a12l users    6 Feb  5 15:38 .ssh
drwxr-xr-x  3 a12l users    4 Dec 25 00:35 .thunderbird
drwxr-xr-x  3 a12l users    3 Jan 26 14:45 .zotero

$ ls -la /persistent/home/a12l/.config/
total 215
drwxr-xr-x 12 a12l users  14 Feb  6 12:53  .
drwxr-xr-x 10 a12l users  13 Feb  6 12:55  ..
drwxr-xr-x  2 a12l users   3 Dec 23 10:41  autostart
drwxr-xr-x  7 a12l users  14 Jan 10 14:42  calibre
drwxr-xr-x 30 a12l users  36 Feb  3 02:25  chromium
drwxr-xr-x  8 a12l users  14 Feb  6 13:42  emacs
drwxr-xr-x  3 a12l users   3 Jan 18 16:14  fontconfig
drwxr-xr-x  3 a12l users   3 Jan 10 18:38  JetBrains
-rw-r--r--  1 a12l users 459 Jan 20 15:20  KeePassXCrc
drwxr-xr-x 10 a12l users  13 Feb  1 22:47 'Mullvad VPN'
drwxr-xr-x  3 a12l users   5 Jan 29 22:31  pijul
drwxr-xr-x  3 a12l users   6 Jan 31 12:40  qBittorrent
-rw-r--r--  1 a12l users 636 Dec 30 17:59  user-dirs.dirs
drwxr-xr-x  8 a12l users  15 Feb  1 15:09  zotero

$ ls -la /persistent/home/a12l/.local/
total 50
drwxr-xr-x  3 a12l users  3 Feb  6 14:00 .
drwxr-xr-x 10 a12l users 13 Feb  6 12:55 ..
drwxr-xr-x  3 a12l users  3 Feb  6 12:55 share

$ ls -la /persistent/home/a12l/.local/share/
total 50
drwxr-xr-x 3 a12l users 3 Feb  6 12:55 .
drwxr-xr-x 3 a12l users 3 Feb  6 14:00 ..
drwxr-xr-x 3 a12l users 9 Jan 18 15:39 fonts

But as you can see below ~/.config; ~/.local; and ~/.local/share has different ownership (root:root) it my $HOME. No problems with the directories and files that I directly list to be handled by Impermanence.

$ ls -la ~/
total 574
drwxr-xr-x 25 a12l users  28 Feb  6 13:44 .
drwxr-xr-x  3 root root    3 Dec 11 14:19 ..
drwxr-xr-x 11 root root   11 Feb  6 13:42 .config
drwxr-xr-x  3 root root    3 Feb  6 13:42 .local
drwxr-xr-x 32 a12l users 422 Feb  4 02:21 Long-Term
drwxr-xr-x  2 a12l users   2 Dec 23 00:35 .minisign
drwxr-xr-x  5 a12l users   5 Jan 16 20:10 .mozilla
drwxr-xr-x  2 a12l users   2 Dec 23 02:17 .scribus
drwxr-xr-x  2 a12l users   6 Feb  5 15:38 .ssh
drwxr-xr-x 18 a12l users 141 Feb  6 13:51 Temporary
drwxr-xr-x  3 a12l users   4 Dec 25 00:35 .thunderbird
drwxr-xr-x  4 a12l users   4 Jan 27 15:57 Zettelkasten
drwxr-xr-x  3 a12l users   3 Jan 26 14:45 .zotero

$ ls -l ~/.config/
total 149
drwxr-xr-x  2 a12l users  3 Dec 23 10:41 autostart
drwxr-xr-x  7 a12l users 14 Jan 10 14:42 calibre
drwxr-xr-x 30 a12l users 36 Feb  3 02:25 chromium
drwxr-xr-x  8 a12l users 14 Feb  6 13:42 emacs
drwxr-xr-x  3 a12l users  3 Jan 18 16:14 fontconfig
drwxr-xr-x  3 a12l users  3 Jan 10 18:38 JetBrains
drwxr-xr-x  3 a12l users  5 Jan 29 22:31 pijul
drwxr-xr-x  3 a12l users  6 Jan 31 12:40 qBittorrent
drwxr-xr-x  8 a12l users 15 Feb  1 15:09 zotero

$ ls -l ~/.local/
total 17
drwxr-xr-x 3 root root 3 Feb  6 13:42 share

$ ls -l ~/.local/share/
total 17
drwxr-xr-x 3 a12l users 9 Jan 18 15:39 fonts

This is the relevant bindmounts

$ findmnt
[...]
├─/home                                  zroot/USERDATA/home                            zfs        rw,relatime,xattr,posixacl
│ └─/home/a12l                           zroot/USERDATA/home/A12L/root                  zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/JetBrains       zroot/HOST/persistent[/home/a12l/.config/JetBrains]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/autostart       zroot/HOST/persistent[/home/a12l/.config/autostart]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/calibre         zroot/HOST/persistent[/home/a12l/.config/calibre]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/chromium        zroot/HOST/persistent[/home/a12l/.config/chromium]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/fontconfig      zroot/HOST/persistent[/home/a12l/.config/fontconfig]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/emacs           zroot/HOST/persistent[/home/a12l/.config/emacs]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/pijul           zroot/HOST/persistent[/home/a12l/.config/pijul]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/qBittorrent     zroot/HOST/persistent[/home/a12l/.config/qBittorrent]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.config/zotero          zroot/HOST/persistent[/home/a12l/.config/zotero]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.local/share/fonts      zroot/HOST/persistent[/home/a12l/.local/share/fonts]
│   │                                                                                   zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.mozilla                zroot/HOST/persistent[/home/a12l/.mozilla]     zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.minisign               zroot/HOST/persistent[/home/a12l/.minisign]    zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.scribus                zroot/HOST/persistent[/home/a12l/.scribus]     zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.ssh                    zroot/HOST/persistent[/home/a12l/.ssh]         zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.zotero                 zroot/HOST/persistent[/home/a12l/.zotero]      zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/.thunderbird            zroot/HOST/persistent[/home/a12l/.thunderbird] zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/Temporary               zroot/USERDATA/home/A12L/temporary             zfs        rw,relatime,xattr,posixacl
│   ├─/home/a12l/Long-Term               zroot/USERDATA/home/A12L/long-term             zfs        rw,relatime,xattr,posixacl
│   └─/home/a12l/Zettelkasten            zroot/USERDATA/home/A12L/zettelkasten          zfs        rw,relatime,xattr,posixacl

And when I look at /run/current-system/activate it seems that the correct arguments is sent to the script :-/

#### Activation script snippet createPersistentStorageDirs:
_localstatus=0
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/autostart' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/calibre' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/chromium' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/emacs' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/fontconfig' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/JetBrains' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.local/share/fonts' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/qBittorrent' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.scribus' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.zotero' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/zotero' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.mozilla' 'a12l' 'users' '0700'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.minisign' 'a12l' 'users' '0700'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/pijul' 'a12l' 'users' '0700'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.ssh' 'a12l' 'users' '0700'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.thunderbird' 'a12l' 'users' '0700'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/var/log' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/var/lib/systemd/coredump' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/etc/NetworkManager/system-connections' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/etc/ssh' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/home/a12l/.config/Mullvad VPN' 'a12l' 'users' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/etc' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/etc/mullvad-vpn' 'root' 'root' '0755'
/nix/store/hkr3fdalaa1pp2rrswrmpqx3dpm2y9c5-impermanence-create-directories '/persistent' '/etc/nix' 'root' 'root' 'u=rwx,g=,o='


if (( _localstatus > 0 )); then
  printf "Activation script snippet '%s' failed (%s)\n" "createPersistentStorageDirs" "$_localstatus"
fi

[talyz]

That's really odd. I can't see why this would happen based on the data you've provided. Is this after a reboot, not just a nixos-rebuild switch?

To get more info on what's happening, I'll have to add some debug printouts in create-directories.

[talyz]

Can you try #85, set enableDebugging = true and report back with the trace from a reboot?

[a12l]

I've done some reorganizing of my filesystem setup. Moved from ZFS to tmpfs, etc. And I no longer experience this problem. I could try to reproduce the problem, or we can close this issue and reopen it again if someone else experience the same problem?

[talyz]

It would be interesting if you could switch back to ZFS to see if that's what causes the issue. It seems to work just fine with Btrfs and tmpfs, at least.

[shadowrylander]

Okay! I believe I can help with this one; I'm on a ZFS root as well, and am being affected by this issue. Anything I can do to help?

[talyz]

Yes! If you can set enableDebugging and report back with a trace from a boot which was affected by this, that would help. This makes it sound more likely to be a ZFS bug, but it would be nice to confirm that the script is doing the right thing.

[shadowrylander]

Where do I set enableDebugging, again?

[talyz]

It's available at the top level of the submodule, so

{
  environment.persistence."/persistent" = {
    enableDebugging = true;
    directories = [
      # ...
    ];
    files = [
      # ...
    ];
  };
}

[shadowrylander]

Right; but how do I trace the reboot too? 😅

[shadowrylander]

Actually, I used journalctl -b, and got a wall of text; do I give you everything, or just until it sets up /etc?

[shadowrylander]

Actually, here's the entire log: https://github.com/shadowrylander/shadowrylander/blob/main/tmp

[talyz]

Yeah, the journalctl -b output is what I want - thanks! Which directories have incorrect ownership for you?

[shadowrylander]

No problem! And same as OP: my user directory. Can't even install anything! 😹

[talyz]

Well, @a12l's issue was that the ownership of ~/.config and some other directories didn't propagate from persistent storage. If that's what you're seeing, but for the root of your home directory, then it doesn't seem to be our bug: from your log I can see that

chown --reference=/persist/home/shadowrylander /home/shadowrylander/

is run many times (should be once for every directory listed under it).

[shadowrylander]

I believe it is; I've got a lot of directories listed there. The issue is that both /home and /home/shadowrylander itself is owned by root, while only the former is meant to be. I don't know how to fix this short of doing it manually with a systemd service.

[talyz]

Who's the owner of /persist/home/shadowrylander?

[shadowrylander]

Another user, curtis; but the owner of /persist/home itself is shadowrylander.

[shadowrylander]

No, wait; switch them around: shadowrylander owns /persist/home/shadowrylander, while curtis owns /persist/home.

[talyz]

In that case, running

chown --reference=/persist/home/shadowrylander /home/shadowrylander/

should make /home/shadowrylander owned by shadowrylander, and that command is run many times on boot according to the log you linked. Could you try switching your / to tmpfs and see if that solves this issue for you, too? If it does, I think we can pretty confidently say that ZFS is to blame.

[shadowrylander]

Hmm... I'll have to figure out the partitioning for that... Be back in a bit.

[talyz]

If you're using the "rollback to empty" trick, it should be as easy as commenting out your normal / filesystem definition and adding

{
  fileSystems."/" = {
    device = "none";
    fsType = "tmpfs";
  };
}

[shadowrylander]

Got it; should I also link another boot trace?

[shadowrylander]

Update: still happening; /etc/fstab shows root on tmpfs, with everything else the same.

[shadowrylander]

Note, however, that my home directories are ZFS datasets as well; should I change them over to tmpfs as well?