[WIP] Allow youki to run with podman in rootless

[orimanabu]

This is a work in progress PR addressing #719. I created this for discussion purpose, please don't merge.

This patch resolves the initial error (Error: failed to create directory /run/youki), but still does not allow youki to run rootless in Podman.

Now I got another error: Error: failed to create session dbus client

$ podman --runtime /home/ori/devel/src/github.com/orimanabu/youki/youki run --userns=auto --name hello hello
[DEBUG crates/youki/src/main.rs:92] 2022-09-04T12:10:03.798359979+09:00 started by user 0 with ArgsOs { inner: ["/home/ori/devel/src/github.com/orimanabu/youki/youki", "delete", "--force", "50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7"] }
[DEBUG crates/youki/src/commands/delete.rs:8] 2022-09-04T12:10:03.798781906+09:00 start deleting 50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7
[DEBUG crates/libcontainer/src/container/container_delete.rs:36] 2022-09-04T12:10:03.799043441+09:00 container status: Stopped
[DEBUG crates/libcontainer/src/container/container_delete.rs:42] 2022-09-04T12:10:03.799120496+09:00 config: YoukiConfig { hooks: None, cgroup_path: "user.slice:libpod:50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7" }
[DEBUG crates/libcontainer/src/container/container_delete.rs:45] 2022-09-04T12:10:03.799144175+09:00 remove dir "/run/user/1000/youki/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7"
Error: failed to delete container 50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7

Caused by:
    container state does not contain cgroup manager
ERRO[0000] Removing container 50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7 from runtime after creation failed 
Error: OCI runtime error: /home/ori/devel/src/github.com/orimanabu/youki/youki: [DEBUG crates/youki/src/main.rs:92] 2022-09-04T12:10:03.736189722+09:00 started by user 0 with ArgsOs { inner: ["/home/ori/devel/src/github.com/orimanabu/youki/youki", "--systemd-cgroup", "create", "--bundle", "/home/ori/.local/share/containers/storage/overlay-containers/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7/userdata", "--pid-file", "/run/user/1000/containers/overlay-containers/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7/userdata/pidfile", "50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7"] }
[DEBUG crates/libcontainer/src/container/init_builder.rs:100] 2022-09-04T12:10:03.740451839+09:00 container directory will be "/run/user/1000/youki/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7"
[DEBUG crates/libcontainer/src/container/container.rs:192] 2022-09-04T12:10:03.740514780+09:00 Save container status: Container { state: State { oci_version: "v1.0.2", id: "50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7", status: Creating, pid: None, bundle: "/home/ori/.local/share/containers/storage/overlay-containers/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7/userdata", annotations: Some({}), created: None, creator: None, use_systemd: None }, root: "/run/user/1000/youki/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7" } in "/run/user/1000/youki/50cd9d798f3f2a2e496e503c4012e7fabe22e69b1bfa29c96245bf1b8b4ec5e7"
[DEBUG crates/libcontainer/src/rootless.rs:40] 2022-09-04T12:10:03.740661343+09:00 rootless container should be created
[INFO crates/libcgroups/src/common.rs:247] 2022-09-04T12:10:03.740881325+09:00 systemd cgroup manager with system bus false will be used
[INFO crates/libcgroups/src/common.rs:247] 2022-09-04T12:10:03.741338885+09:00 systemd cgroup manager with system bus false will be used
Error: failed to create session dbus client

Caused by:
    0: failed to create container
    1: failed to create session dbus client
    2: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

It seems that CgroupPath in config.json is empty, I'll look into it later.

$ podman inspect hello | jq '.[].State'
{
  "OciVersion": "1.0.2-dev",
  "Status": "created",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 0,
  "Error": "",
  "StartedAt": "0001-01-01T00:00:00Z",
  "FinishedAt": "0001-01-01T00:00:00Z",
  "Health": {
    "Status": "",
    "FailingStreak": 0,
    "Log": null
  },
  "CheckpointedAt": "0001-01-01T00:00:00Z",
  "RestoredAt": "0001-01-01T00:00:00Z"
}

Signed-off-by: Manabu Ori [email protected]

[codecov-commenter]

Codecov Report

Merging #1171 (b567986) into main (3a29e2d) will decrease coverage by 0.04%. The diff coverage is 28.57%.

@@            Coverage Diff             @@
##             main    #1171      +/-   ##
==========================================
- Coverage   69.41%   69.37%   -0.05%     
==========================================
  Files         118      118              
  Lines       12446    12453       +7     
==========================================
- Hits         8640     8639       -1     
- Misses       3806     3814       +8     

[utam0k]

Hi, @orimanabu I made this PR draft because you put WIP on the title. If this PR is ready for PR, please feel free to make it ready for review.

[riyad]

FYI: I tried out these changes and it's goint in the right direction, but fails with the following error now:

Error: rootless container requires valid user namespace definition

Adding a few dbg!()s produced:

$ podman run --runtime $PWD/youki --rm alpine
[crates/youki/src/main.rs:144] getuid().as_raw() = 0
[crates/youki/src/main.rs:146] root_path = None
[crates/libcgroups/src/common.rs:228] content = "         0       1000          1\n         1     100000      65536\n"
[crates/youki/src/main.rs:154] rootless_required() = true
[crates/youki/src/main.rs:162] Path::new(&path).join("youki") = "/run/user/1000/youki"
[ERROR crates/youki/src/main.rs:138] 2022-11-07T21:29:36.138970068+01:00 error in executing command: failed to delete container 5affe9849620e2dfaec2f371d59d8257bead1cf97a27126b6f9f826b32204a14

Caused by:
    0: failed to load runtime spec for container 5affe9849620e2dfaec2f371d59d8257bead1cf97a27126b6f9f826b32204a14
    1: No such file or directory (os error 2)
Error: failed to delete container 5affe9849620e2dfaec2f371d59d8257bead1cf97a27126b6f9f826b32204a14

Caused by:
    0: failed to load runtime spec for container 5affe9849620e2dfaec2f371d59d8257bead1cf97a27126b6f9f826b32204a14
    1: No such file or directory (os error 2)
ERRO[0000] Removing container 5affe9849620e2dfaec2f371d59d8257bead1cf97a27126b6f9f826b32204a14 from runtime after creation failed
Error: OCI runtime error: /home/riyad/src/youki/youki: [crates/youki/src/main.rs:144] getuid().as_raw() = 0
[crates/youki/src/main.rs:146] root_path = None
[crates/libcgroups/src/common.rs:228] content = "         0       1000          1\n         1     100000      65536\n"
[crates/youki/src/main.rs:154] rootless_required() = true
[crates/youki/src/main.rs:162] Path::new(&path).join("youki") = "/run/user/1000/youki"
[crates/libcgroups/src/common.rs:228] content = "         0       1000          1\n         1     100000      65536\n"
[ERROR crates/youki/src/main.rs:138] 2022-11-07T21:29:36.121815933+01:00 error in executing command: rootless container requires valid user namespace definition
Error: rootless container requires valid user namespace definition

[YJDoc2]

Hey @orimanabu, are you still working on this? If you are busy with something else, and currently cannot work on this, I might use this PR as guide and try to make the other changes required to make it work. One of the concerns I have for this PR is that , there might have been a lot of changes in the files since you opened this , so you might get conflicts once you un-WIP this.

Let us know if you are still working on this, and need any help, I'll be happy help as much as I can.

Thanks :)

[utam0k]

Hi, @orimanabu. Sorry if I bother you, but this is kindly reminder. We would like to know if you have enough time to handle it. May I ask you to tell me? It is most important that you develop youki with a lot of fun. Please feel free to ask me for help ;)

[orimanabu]

Hi, I'm very sorry to leave this PR open for so long. Please go ahead and feel free to use this PR or another to proceed.

I hope I'll have spare time in winter vacation! Now I have a new issue regarding podman + youki in which rootful podman run fails. I'll make another issue about it.

[YJDoc2]

No worries. I'll try to use this and what you have implemented, and work towards getting youki to work with podman rootless. Thanks a lot for starting this and building a road to follow :) I'll keep this PR open, until I / someone opens another one that supersedes this.

[orimanabu]

Hi, I've just rebased and investigated a little bit further. The error message is:

$ podman unshare ./youki create test
[ERROR crates/youki/src/main.rs:138] 2023-01-18T22:56:20.672293762+09:00 error in executing command: failed to create session dbus client

Caused by:
    0: failed to create container
    1: failed to create session dbus client
    2: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Error: failed to create session dbus client

Caused by:
    0: failed to create container
    1: failed to create session dbus client
    2: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Youki seemed to talk to dbus session in Manager::new() then failed.

My guess is dbus authorization mechanism rejected youki.

The reason is that... According to ^1 and ^2, dbus uses UID to verify the user connecting. When a normal user (in my env, the user name is ori and UID is 1000) runs youki via rootless podman, youki runs as UID 0 (root) in a user namespace podman created.

Dbus-broker for my user ori's session runs as UID 1000, but youki connected as UID 0 (because it runs as UID 0 in podman's user namespace), so dbus broker rejected youki.

I confirmed that with strace.

1. rootless youki without podman:

[ori@localhost youki]$ strace -e sendto -o strace.log ./youki create test2
[WARN crates/libcontainer/src/process/container_init_process.rs:90] 2023-01-18T14:32:55.347765209+00:00 masked path "/proc/timer_stats" not exist
[WARN crates/libcontainer/src/process/container_init_process.rs:90] 2023-01-18T14:32:55.347800321+00:00 masked path "/proc/sched_debug" not exist

[ori@localhost youki]$ cat strace.log 
sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "AUTH EXTERNAL 31303030\r\n", 24, MSG_NOSIGNAL, NULL, 0) = 24
sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19
sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=381820, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

The argument of AUTH dbus method call is ASCII 31303030 which means UID 1000, and youki create succeeded.

2. rootless youki via podman:

[ori@localhost youki]$ podman unshare strace -e sendto -o strace.log ./youki create test2
[ERROR crates/youki/src/main.rs:138] 2023-01-18T23:35:17.258757674+09:00 error in executing command: failed to create session dbus client

Caused by:
    0: failed to create container
    1: failed to create session dbus client
    2: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Error: failed to create session dbus client

Caused by:
    0: failed to create container
    1: failed to create session dbus client
    2: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

[ori@localhost youki]$ cat strace.log
sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
+++ exited with 1 +++

youki in podman user namespace tried to connect as UID 0 (ASCII 30) and failed.

I also confirm that from dbus-broker's side. strace on dbus-broker contains below calls in which it rejects from UID 0 (ASCII 30).

recvmsg(113, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0AUTH EXTERNAL 30\r\n", iov_len=2048}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 19
sendmmsg(113, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="REJECTED EXTERNAL\r\n", iov_len=19}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=19}], 1, MSG_DONTWAIT|MSG_NOSIGNAL) = 1

So the counterplan is... We should not use dbus when run in rootless UID 0 ? I'll look into runc and crun how they handle that in rootless mode.

[utam0k]

@orimanabu Thanks a lot! I also have interested in the behavior of other runtimes.

Hi @Furisto Do you know anything about this issue?

[YJDoc2]

Hey, So I did a bit of digging, and found out this :

• As orimanabu have correctly noted, the issue here is while doing authentication to dbus, dbus-rs uses uid 0, because we are running inside the new userspace. However, our actual uid (as we are running non-root) is non-0 (eg. 1000) hence the auth fails.
• I checked runc, and essentially what it does to to implement dbus management itself see this . In here, it calls this which detects the processes "actual" uid by running busctl command and parsing the o/p to get OwnerUID.
• I checked dbus-rs, and it does not provide any way to pass uid for opening channel. Further, this is not simply an issue of adding new api, because dbus-rs itself uses a call dbus_bus_get_private which directly calls ffi and creates a connection. In fact even runc uses two different libraries to work with dbus while doing a custom authentication.
• It appears that if we want to give a custom uid for dbus connection, we will have to (directly or indirectly) own a lot of dbus connection and interfacing code instead of simply using the library.

[utam0k]

@YJDoc2 To be honest, I am not very familiar with this part. How about we start an issue on this and see how the community https://github.com/diwic/dbus-rs reacts to this problem? By the way, is this a problem that can be solved by forking and improving it?

[YJDoc2]

Hey, I'll open an issue on dbus-rs and see how that goes. I'm not sure if simply forking and doing minor code changes would solve this as this might require some fundamental changes to how the library connects with dbus. Maybe the dbus community might have a better idea and help us out. I'll keep this updated as things go on. Thanks!

[YJDoc2]

Closing as completed via #2370 , and released in v0.3.0 :tada: Thank you :pray: