Cachepot fails to start server unless run as root

[LukeMathWalker]

Hey! I've been trying to introduce cachepot into our CI pipelines (CircleCI) and I am having some issues that I believe to be related to permissions. If cachepot --start-server is executed without using sudo, the command always fails complaining that it timed out waiting for the server to start up. If we run it with sudo, the server starts up correctly but we then run into issues when interacting with cargo.

What could be needing elevated permissions that causes cachepot --start-server to fail (unfortunately without emitting any useful log even at trace level)?

[drahnr]

We are working on #128 for the dist case which only require user namespaces. We would need additional info to understand if your issue would be addressed by this, or if you're running the client side compilation. Could you show which binary you use and how you compile cachepot?

We'd need the following env in the CI env and cachepot --version besides the way you compiled the binary.

Thanks

[LukeMathWalker]

We built cachepot using cargo install --locked --git https://github.com/paritytech/cachepot. We are using the local cache.

[drahnr]

Can you provide the env vars? Also the logs, even if unconclusive to you, it still could narrow down the issue.

[LukeMathWalker]

Sure! Environment variables:

      RUSTC_WRAPPER: "/usr/local/cargo/bin/cachepot"
      CACHEPOT_DIR: "~/.cache/cachepot"
      CACHEPOT_ERROR_LOG: "/tmp/cachepot_log.txt"
      CACHEPOT_LOG: "trace"
      CACHEPOT_IDLE_TIMEOUT: 0
      CACHEPOT_SERVER_PORT: 8001

Logs when running cachepot --start-server:

[2021-12-17T11:13:15Z TRACE cachepot::cmdline] parse
cachepot 0.1.0
[2021-12-17T11:13:15Z TRACE cachepot::cmdline] parse
[2021-12-17T11:13:15Z DEBUG cachepot::config] Attempting to read config file at "/home/********/.config/cachepot/config"
[2021-12-17T11:13:15Z DEBUG cachepot::config] Couldn't open config file: failed to open file `/home/********/.config/cachepot/config`
[2021-12-17T11:13:15Z INFO  cachepot::config] Using the default configuration.
[2021-12-17T11:13:15Z TRACE cachepot::commands] Command::StartServer
cachepot: Starting the server...
[2021-12-17T11:13:15Z TRACE cachepot::commands] run_server_process
[2021-12-17T11:13:15Z TRACE mio::poll] registering event source with poller: token=Token(0), interests=READABLE | WRITABLE
[2021-12-17T11:13:15Z TRACE mio::poll] registering event source with poller: token=Token(1), interests=READABLE | WRITABLE
[2021-12-17T11:13:15Z TRACE cachepot::cmdline] parse
[2021-12-17T11:13:15Z DEBUG cachepot::config] Attempting to read config file at "/home/********/.config/cachepot/config"
[2021-12-17T11:13:15Z DEBUG cachepot::config] Couldn't open config file: failed to open file `/home/********/.config/cachepot/config`
[2021-12-17T11:13:15Z INFO  cachepot::config] Using the default configuration.
[2021-12-17T11:13:15Z TRACE cachepot::commands] Command::InternalStartServer
[2021-12-17T11:13:25Z TRACE mio::poll] deregistering event source from poller
cachepot: error: Timed out waiting for server startup

[drahnr]

I think we have to add additional logs, this is inconclusive. Note that we had a rather large rename to avoid the re-use of server for multiple things across the code base, this includes the cli args, so you'd have to use --start-coordinator with the latest git.

[Xanewok]

@LukeMathWalker could you try again, since #128 is merged? With unprivileged user namespaces + overlayfs support (make sure you're running 5.11+ kernel) you don't need to be root.

Make sure you're running with CACHEPOT_SANDBOX env var set to userns (this feature is unstable for now)