WIP: feat(core): Add `withMountedSSH` API to `Container`

[grouville]

Creates a withMountedSSH API to core.Container

• Partially solves: https://github.com/dagger/dagger/issues/3581
• Forwards SSH_AUTH_SOCK to forwards ssh-agent connection

Both core.Build and core.Exec might benefit from it

Tests are on the way

wdyt @vito ?

TODO: add tests + go generate

Signed-off-by: Guillaume de Rouville

[aluzzardi]

[this is just my opinion, @vito to decide]

I think we could fix this AND #3712 (docker socket) in one shot.

Suggestion:

• container.WithMountedSocket()
• host.Socket()

Example:

• SSH Agent forwarding
  • container.WithMountedSocket("/ssh-agent.sock", c.Host().Socket(os.Getenv("SSH_AUTH_SOCK")))
• Docker forwarding
  • container.WithMountedSocket("/var/run/docker.sock", c.Host().Socket("/var/run/docker.sock"))

Bikeshedding:

• In the future, we could use the same mechanism to forward TCP connections to/from the container (using the shim + engine sessions)
  • Especially useful with services
  • Bikeshedding: WithMountedUnixSocket to leave space for WithTCPSocket etc?

/cc @shykes

[shykes]

[this is just my opinion, @vito to decide]

I think we could fix this AND #3712 (docker socket) in one shot.

Yes please, no ssh-specific option please. That would be a step back from the 0.2 API which correctly generalized sockets.

[shykes]

IMO before we merge this, we need to lock down the API design for a broader set of features, even if we don't implement all those features right away. At the very least we should make sure we use the lessons from the 0.2 API design (both the good and the bad).

Some features that we should design before implementing any of them:

• container-to-host: processes running inside the container can connect to a tcp port, udp port or unix socket, and the traffic is forwarded to a given address in the host network stack
  • Example 1: docker CLI inside the container can connect to host's docker engine
  • Example 2: mysql client configured to connect to tcp://localhost:mysql can seamlessly connect to a DB on the host, without changing its configuration
• host-to-container: processes running inside the container can listen on a tcp, udp or unix address, and traffic is forwarded from a given listening address on the host
  • Example: run a web service in a pipeline, connect to it from the browser on https://localhosts:8080
• Container-to-container: process in container A can connect to container B
  • Example: frontend service is configured to connect to tcp://localhost:9999 and udp://localhost:1000, those ports can be forwarded to the backend service without changing the service configuration

Again: we don't need to implement all these features now (or even soon), but we should discuss the API design for them as a whole.

cc @aluzzardi @vito @sipsma

[grouville]

@shykes following your comment, not sure about the next actionable step? Shall I:

• close that PR ?
• make an issue out-of-it ?
• Implement @aluzzardi's proposal (under supervision of @vito), or let Alex do it ?

This feature might unlock very painful use-cases -> impossible to git push from an exec step using ssh repo urls. We need to create a PAT, have a remote HTTP URL, set the GIT_ASKPASS env var and have a script showing the PAT, all inside the same container

[grouville]

Ping: one of our power user just requested this feature -> https://discord.com/channels/707636530424053791/1040217609750188032/1040217609750188032

[aluzzardi]

Ping: one of our power user just requested this feature ->

Yeah, and we also kinda need this to be able to run full tests using dagger (to test dagger provisioning in dagger, we need the executed container to be able to launch a new engine). /cc @sipsma

IMO before we merge this, we need to lock down the API design for a broader set of features, even if we don't implement all those features right away.

That's pretty hard. I'll take a stab at it (very incomplete)

container-to-host: processes running inside the container can connect to a tcp port, udp port or unix socket, and the traffic is forwarded to a given address in the host network stack

2 parts:

• Host() API exposes unix/tcp/udp sockets
• Container() API allows to pass unix/tcp/udp sockets

c.Host().UnixSocket(path string) *dagger.Socket
c.Host().TCPSocket(port int) *dagger.Socket
c.Host().UDPSocket(port int) *dagger.Socket

container.WithUnixSocket(path string, socket *dagger.Socket)
container.WithTCPSocket(port int, socket *dagger.Socket)
container.WithUDPSocket(port int, socket *dagger.Socket)

Usage:

container.WithUnixSocket("/var/run/docker.sock", c.Host().UnixSocket("/var/run/docker.sock"))

Under the hood, it's all unix sockets (because it's all we have from buildkit:

• Container.With{TCP,UDP}Socket is handled by the shim (unix <-> tcp/udp proxy). Q: does udp over unix even make sense?
• Host().With{Unix,TCP,UDP}Socket is handled by the engine session
  • Unix is a passthrough
  • TCP / UDP is a unix proxy (same question about udp over unix)

Container-to-container: process in container A can connect to container B

Does this make sense for anything but services?

Regular Execs are:

• short lived
• eventual (when do you connect to the socket? might be right away, might be in 10 mins)
• cached (you might never be able to connect if the Exec is cached)

In that case, mirror the Host() API in Service():

c.Container().WithTCPSocket(8080, myService.TCPSocket(8080))

I know @vito did a lot of this with bass. There's a lot of considerations to take into account, especially around not busting the cache.

host-to-container: processes running inside the container can listen on a tcp, udp or unix address, and traffic is forwarded from a given listening address on the host

Same question as above -- does it make sense for non-services? I think not, for the same reasons.

However, even if limited to services, the tricky part here is how are ports exposed to the host?

• One solution is to rely on the CLI, in the same fashion as dagger attach for services TTY
  • e.g. dagger service proxy <id> <local port>:<service port>
  • Handled as a GraphQL WS endpoint
• Another solution is to expose this in the SDK, but it's hard
  • e.g. c.Host().ProxySocket(*dagger.Socket)
  • But this is not GraphQL -- it can't be codegen'd, needs special handling
  • Could be implemented by each SDK over WS
  • Or could be implemented over the gRPC transport of sessions, but, TBD

/cc @vito @sipsma

[mircubed]

When this PR merges, we need to alert this user -https://discord.com/channels/707636530424053791/1042013918756868126

[shykes]

@aluzzardi @grouville I opened an issue to unblock us: https://github.com/dagger/dagger/issues/3850

[shykes]

I propose closing this, and continuing all discussion in https://github.com/dagger/dagger/issues/3850 for easier discoverability.