/run/user/$UID/doc not mounted automatically when pam_namespaces is used

[Erick555]

While using polyinstantiated /tmp and /var/tmp dirs through pam_namespace module running flatpak apps fail with:

flatpak run --verbose --command=sh org.freedesktop.Platform//18.08
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Regenerating ld.so.cache /home/user/.cache/flatpak/ld.so/422e06976b310760619b088995f5189300c93f8a26b374e2f91ab50e9e8f667f
F: Running: 'bwrap --args 19 ldconfig -X -C /run/ld-so-cache-dir/422e06976b310760619b088995f5189300c93f8a26b374e2f91ab50e9e8f667f'
F: Cleaning up unused container id 3362085733
F: Allocated instance id 1139753162
F: Add defaults in dir /org/freedesktop/Platform/
F: Add locks in dir /org/freedesktop/Platform/
F: Disallowing ipc access
F: Disallowing network access
F: Running 'bwrap --args 32 xdg-dbus-proxy --args=34'
F: Running 'bwrap --args 32 sh'
bwrap: Can't find source path /run/user/1000/doc/by-app/org.freedesktop.Platform: No such file or directory

This is because /run/user/1000/doc fuse mount point doesn't exist after xdg-document-portal started. Restarting it manually temporarily fixes the issue:

 env G_MESSAGES_DEBUG=all /usr/lib/xdg-document-portal --replace --verbose
XDP: org.freedesktop.portal.Documents acquired
XDP: xdp_fuse_getattr 1 (fi=(nil))
XDP: xdp_fuse_getattr <- OK
XDP: xdp_fuse_lookup 1/by-app -> 
XDP: xdp_fuse_lookup <- inode 2
XDP: xdp_fuse_lookup 2/org.freedesktop.Platform -> 
XDP: xdp_fuse_lookup <- inode 3
XDP: xdp_fuse_getattr 1 (fi=(nil))
XDP: xdp_fuse_getattr <- OK
XDP: xdp_fuse_access 1 0
XDP: xdp_fuse_access 1 0
XDP: xdp_fuse_access 1 0
XDP: xdp_fuse_access 1 0

/run/user/1000/doc /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100

The same problem is reported elsewhere although caused by something else: https://github.com/flatpak/flatpak/issues/1359 https://github.com/flatpak/xdg-desktop-portal/issues/305

[Erick555]

Running /usr/lib/xdg-document-portal automatically at session start outside systemd --user service works as well.

[matthiasclasen]

whats the connection between pam_namespace being used and /run/user/1000/doc not existing ?

[matthiasclasen]

Alex, can you have a look at this? Might be nice to fix for 1.4

[Erick555]

whats the connection between pam_namespace being used and /run/user/1000/doc not existing ?

No idea but disabling pam_namespace make it work again. I guess there is some fragility in creating /run/user/1000/doc mountpoint as it fails in many different circumstances.

[Erick555]

I can add that I'm using KDE Plasma 5.15.5 session. It fails for both Xorg and Wayland. Interestingly even adding:

ExecStart=/usr/lib/xdg-document-portal --verbose
Environment=G_MESSAGES_DEBUG=all
StandardOutput=file:/home/user/.cache/xdg-doc.log
StandardError=inherit

to xdg-document-portal.service doesn't produce any output thus it's hard to say what's going on there.

[alexlarsson]

I think the problem might be with mount propagation. With pam_namespaces various processes will run in different namespaces, and if you mount something in one of them they need to propagate to the other for them to be visible there. Clearly the fuse mountpoint is mounted in one namespace, but not reaching the one the app is running in.

What happens normally is that flatpak at startup calls a dbus call org.freedesktop.portal.Documents.GetMountPoint() on the session bus. This will cause dbus-daemon to autostart the portal and it will mount the fuse mount. However, this will be the namespace of dbus, rather than the one flatpak is running in. Typically they will be the same, but maybe pam_namespaces is affecting one of these making it different.

Now, that by itself is not necessarily problematic, due to mount propagation. What we want is that the mount by the portal should be propagated up to the closes shared namespace, and then down into the namespace that is running the flatpak. The basic requirement is that there is a basic global namespace for the entire login session, and that the namespaces are set up such that mounts propagate up from the portal namespace and down everywhere.

The pam_namespace page (https://linux.die.net/man/8/pam_namespace) says this:

The pam_namespace module disassociates the session namespace from the parent namespace. Any mounts/unmounts performed in the parent namespace, such as mounting of devices, are not reflected in the session namespace. To propagate selected mount/unmount events from the parent namespace into the disassociated session namespace, an administrator may use the special shared-subtree feature. For additional information on shared-subtree feature, please refer to the mount(8) man page and the shared-subtree description at http://lwn.net/Articles/159077 and http://lwn.net/Articles/159092.

This points to the problem. I honestly don't really know how pam_namespace works, so I can't tell how the tree of namespaces look here. If pam_namespace only creates one namespace as per the above per session and then everything ends up in it then it should be fine. However, if (say) services spawned from dbus-daemon gets their own namespace then it is not going to work. I think you need to figure out how your setup ends up.

What does the output of sudo lsns -t mnt --list --output-all look like?

[Erick555]

Without pam_namespace (some flatpak app runs):

        NS TYPE PATH               NPROCS   PID  PPID COMMAND                                             UID USER             NETNSID NSFS
4026531840 mnt  /proc/1/ns/mnt        150     1     0 /sbin/init                                            0 root                     
4026531860 mnt  /proc/32/ns/mnt         1    32     2 kdevtmpfs                                             0 root                     
4026532211 mnt  /proc/546/ns/mnt        1   546     1 /usr/lib/systemd/systemd-udevd                        0 root                     
4026532253 mnt  /proc/754/ns/mnt        1   754     1 /usr/lib/systemd/systemd-timesyncd                  980 systemd-timesync         
4026532391 mnt  /proc/941/ns/mnt        2   941     1 /usr/bin/dbus-broker-launch --scope system --audit   81 dbus                     
4026532394 mnt  /proc/1032/ns/mnt       1  1032     1 /usr/lib/systemd/systemd-logind                       0 root                     
4026532395 mnt  /proc/1035/ns/mnt       2  1035     1 /usr/bin/NetworkManager --no-daemon                   0 root                     
4026532397 mnt  /proc/17859/ns/mnt      1 17859 17858 xdg-dbus-proxy --args=34                           1000 user                       
4026532398 mnt  /proc/17862/ns/mnt      2 17862 17849 bwrap --args 32 sh                                 1000 user                       
4026532436 mnt  /proc/1366/ns/mnt       1  1366     1 /usr/bin/iio-sensor-proxy                             0 root                     
4026532471 mnt  /proc/2042/ns/mnt       1  2042     1 /usr/lib/upowerd                                      0 root                     

With pam_namespace:

        NS TYPE PATH               NPROCS   PID PPID COMMAND                                             UID USER             NETNSID NSFS
4026531840 mnt  /proc/1/ns/mnt        105     1    0 /sbin/init                                            0 root                     
4026531860 mnt  /proc/32/ns/mnt         1    32    2 kdevtmpfs                                             0 root                     
4026532211 mnt  /proc/546/ns/mnt        1   546    1 /usr/lib/systemd/systemd-udevd                        0 root                     
4026532253 mnt  /proc/754/ns/mnt        1   754    1 /usr/lib/systemd/systemd-timesyncd                  980 systemd-timesync         
4026532391 mnt  /proc/941/ns/mnt        2   941    1 /usr/bin/dbus-broker-launch --scope system --audit   81 dbus                     
4026532394 mnt  /proc/1032/ns/mnt       1  1032    1 /usr/lib/systemd/systemd-logind                       0 root                     
4026532395 mnt  /proc/1035/ns/mnt       2  1035    1 /usr/bin/NetworkManager --no-daemon                   0 root                     
4026532397 mnt  /proc/18103/ns/mnt     19 18103    1 /usr/lib/systemd/systemd --user                    1000 user                       
4026532398 mnt  /proc/17995/ns/mnt     41 17995    1 login -- user                                           0 root                     
4026532436 mnt  /proc/1366/ns/mnt       1  1366    1 /usr/bin/iio-sensor-proxy                             0 root                     
4026532471 mnt  /proc/2042/ns/mnt       1  2042    1 /usr/lib/upowerd                                      0 root    

It seems to me that systemd --user instance ends up in different namespace than login process, right?

[alexlarsson]

Yes, and you expect some stuff to be in a different namespace for this to work, so systemd --user being the root for that sounds good to me. As long as all apps in the session are in that runtime (or, as it will be for the bwrap instance, a sub-namespace of it) then everything should be ok.

However, if some other process in the seesion is in the root namespace (in particular session dbus activated things like the document portal) then things will break. So, i guess the next thing to look at is lsns -t mnt -p $pid of various processes (comparing it to the first column in the lsns output) to see e.g. where the document portal and the flatpak app ends up.

If you list things with lsns -t mnt --output NS,TYPE,NPROCS,PPID,PPID,USER,COMMAND you also get the parent pid of the namespace root so you can figure out which namespaces inherit from which.

Another thing we need to look at is the mount options of the document portal fuse filesystem in various processes. For example, on my system some arbitrary pid in the session has these options:

$ grep /run/user/.*/doc /proc/self/mountinfo
1413 1353 0:55 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:775 - fuse /dev/fuse rw,user_id=1000,group_id=1000

Note the shared:775 part here, which is the interesting part. Shared means it will have exactly the same mounts as its parent namespace (and all other namespaces where it shares this group 775).

Compare this to a flatpak:ed process:

$ flatpak run --command=grep org.gnome.gedit  /run/user/.*/doc /proc/self/mountinfo
1476 1343 0:55 /by-app/org.gnome.gedit /run/user/1000/doc rw,nosuid,nodev,relatime master:775 - fuse /dev/fuse rw,user_id=1000,group_id=1000

Here we see instead "master:775". This means the mount is in a slave state. Any changes to the global group (775) will be propagated into this namespace, but any changes we do will not get propagated out.

[Erick555]

@alexlarsson Situation after xdg-document-portal is started through systemd:

$ sudo lsns -t mnt --list --output-all
        NS TYPE PATH               NPROCS   PID  PPID COMMAND                                             UID USER             NETNSID NSFS
4026531840 mnt  /proc/1/ns/mnt        104     1     0 /sbin/init                                            0 root                     
4026531860 mnt  /proc/32/ns/mnt         1    32     2 kdevtmpfs                                             0 root                     
4026532211 mnt  /proc/537/ns/mnt        1   537     1 /usr/lib/systemd/systemd-udevd                        0 root                     
4026532351 mnt  /proc/1154/ns/mnt       1  1154     1 /usr/lib/systemd/systemd-timesyncd                  980 systemd-timesync         
4026532391 mnt  /proc/1205/ns/mnt       2  1205     1 /usr/bin/dbus-broker-launch --scope system --audit   81 dbus                     
4026532392 mnt  /proc/1207/ns/mnt       1  1207     1 /usr/bin/NetworkManager --no-daemon                   0 root                     
4026532393 mnt  /proc/1208/ns/mnt       1  1208     1 /usr/bin/irqbalance --foreground                      0 root                     
4026532394 mnt  /proc/1214/ns/mnt       1  1214     1 /usr/lib/systemd/systemd-logind                       0 root                     
4026532398 mnt  /proc/34823/ns/mnt     13 34823     1 /usr/lib/systemd/systemd --user                    1000 user1                       
4026532399 mnt  /proc/34720/ns/mnt     47 34720     1 login -- user1                                           0 root                     
4026532470 mnt  /proc/2045/ns/mnt       1  2045     1 /usr/lib/upowerd                                      0 root                     

lsns -t mnt -p (pid of xdg-document-portal)
        NS TYPE NPROCS   PID USER COMMAND
4026532398 mnt      18 34823 user1   /usr/lib/systemd/systemd --user
lsns -t mnt -p (pid of xdg-permission-store)
        NS TYPE NPROCS   PID USER COMMAND
4026532398 mnt      18 34823 user1   /usr/lib/systemd/systemd --user
lsns -t mnt -p (pid of flatpak-session-helper)
        NS TYPE NPROCS   PID USER COMMAND
4026532398 mnt      18 34823 user1   /usr/lib/systemd/systemd --user
sudo lsns -t mnt -p (pid of flatpak-system-helper)
        NS TYPE NPROCS PID USER COMMAND
4026531840 mnt      94   1 root /sbin/init

Situation after xdg-document-portal is started manually:

$ sudo lsns -t mnt --list --output-all
        NS TYPE PATH               NPROCS   PID  PPID COMMAND                                             UID USER             NETNSID NSFS
4026531840 mnt  /proc/1/ns/mnt         93     1     0 /sbin/init                                            0 root                     
4026531860 mnt  /proc/32/ns/mnt         1    32     2 kdevtmpfs                                             0 root                     
4026532211 mnt  /proc/537/ns/mnt        1   537     1 /usr/lib/systemd/systemd-udevd                        0 root                     
4026532351 mnt  /proc/1154/ns/mnt       1  1154     1 /usr/lib/systemd/systemd-timesyncd                  980 systemd-timesync         
4026532391 mnt  /proc/1205/ns/mnt       2  1205     1 /usr/bin/dbus-broker-launch --scope system --audit   81 dbus                     
4026532392 mnt  /proc/1207/ns/mnt       1  1207     1 /usr/bin/NetworkManager --no-daemon                   0 root                     
4026532394 mnt  /proc/1214/ns/mnt       1  1214     1 /usr/lib/systemd/systemd-logind                       0 root                     
4026532398 mnt  /proc/34823/ns/mnt     18 34823     1 /usr/lib/systemd/systemd --user                    1000 user1                       
4026532399 mnt  /proc/34720/ns/mnt     42 34720     1 login -- user1                                           0 root                     
4026532400 mnt  /proc/39956/ns/mnt      1 39956 39955 xdg-dbus-proxy --args=34                           1000 user1                       
4026532409 mnt  /proc/39959/ns/mnt      2 39959 39945 bwrap --args 32 sh                                 1000 user1                       
4026532470 mnt  /proc/2045/ns/mnt       1  2045     1 /usr/lib/upowerd                                      0 root                     

$ sudo lsns -t mnt -p (pid of xdg-document-portal)
        NS TYPE NPROCS   PID USER COMMAND
4026532399 mnt      44 34720 root login -- user1

$ sudo lsns -t mnt -p (pid of bwrap)
        NS TYPE NPROCS   PID USER COMMAND
4026532399 mnt      44 34720 root login -- user1 

For me it seems that things doesn't work when xdg-document-portal is started under systemd --user namespace and work when it's started under login -- user1 namespace (PAM).

As for the fuse mounts it shows neither of shared/master mount options when pam_namespace is enabled:

$ grep /run/user/1000/doc /proc/self/mountinfo
1007 974 0:54 / /run/user/1000/doc rw,nosuid,nodev,relatime - fuse /dev/fuse rw,user_id=1000,group_id=100

[📦 org.freedesktop.Platform ~]$ grep /run/user/1000/doc /proc/self/mountinfo
1811 1218 0:54 /by-app/org.freedesktop.Platform /run/user/1000/doc rw,nosuid,nodev,relatime - fuse /dev/fuse rw,user_id=1000,group_id=100

When pam_namespace is disabled it show them with 559 permission set:

$ grep /run/user/1000/doc /proc/self/mountinfo
1460 682 0:76 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:599 - fuse /dev/fuse rw,user_id=1000,group_id=100

[📦 org.freedesktop.Platform ~]$ grep /run/user/1000/doc /proc/self/mountinfo
1970 1923 0:76 /by-app/org.freedesktop.Platform /run/user/1000/doc rw,nosuid,nodev,relatime master:599 - fuse /dev/fuse rw,user_id=1000,group_id=100

[refi64]

Just to be clear I'm noticing two different trends here:

• pam_namespace somehow causing the permissions to get messed up, fix is to restart the portal under another namespace.
• Some state gets messed up (sometimes after suspend?) and the permissions are messed up until the service is restarted.

[eyduh]

I have similar issues running kde plasma under alpine linux with openrc

$ flatpak run --verbose --command=sh org.freedesktop.Platform//21.08
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user1/.local/share/flatpak
F: Opening user flatpak installation at path /home/user1/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user1/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Regenerating ld.so.cache /home/user1/.cache/flatpak/ld.so/b90d73eda21bc7bf4674dd679daf081e2ebeffc165fee967ff3443664283e399
F: /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08/aded519497b3817b395cf872f38719a7afce74ec838acf49fb1502042d19bc3c/files/lib32 does not exist
F: Running: 'bwrap --args 18 ldconfig -X -C /run/ld-so-cache-dir/b90d73eda21bc7bf4674dd679daf081e2ebeffc165fee967ff3443664283e399.iofmo5'
F: /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08/aded519497b3817b395cf872f38719a7afce74ec838acf49fb1502042d19bc3c/files/lib32 does not exist
F: Cleaning up unused container id 2276268734
F: Cleaning up per-app-ID state for com.bitwarden.desktop
F: Allocated instance id 1595076475
F: Add defaults in dir /org/freedesktop/Platform/
F: Add locks in dir /org/freedesktop/Platform/
F: Disallowing ipc access
F: Disallowing network access
F: Failed to run in transient scope: No systemd user session available, cgroups not available
F: Running 'bwrap --args 35 xdg-dbus-proxy --args=37'
F: Running 'bwrap --args 35 sh'
bwrap: Can't find source path /run/user/1000/doc/by-app/org.freedesktop.Platform: No such file or directory

Strange thing is that it works for some flatpaks, note firefox:

$ doas lsns -t mnt --list --output-all
doas (user1@home) password: 
        NS TYPE PATH              NPROCS   PID  PPID COMMAND                    UID USER NETNSID NSFS PNS        ONS
4026531841 mnt  /proc/1/ns/mnt       251     1     0 /sbin/init                   0 root                0 4026531837
4026531862 mnt  /proc/53/ns/mnt        1    53     2 kdevtmpfs                    0 root                0 4026531837
4026532526 mnt  /proc/3373/ns/mnt     14  3373  3354 s6-svscan -t0 /var/run/s6    0 root                0 4026531837
4026532733 mnt  /proc/4969/ns/mnt      1  4969  4968 xdg-dbus-proxy --args=57  1000 user1                 0 4026532630
4026532735 mnt  /proc/4971/ns/mnt     15  4971  4934 bwrap --args 55 firefox   1000 user1                 0 4026532734

(Not sure if related but my flatpak list broke for non-root users at some point after installation, probably due to some permissions error on flatpak directories)