[feat]: Port forwarding to make it possible to route non-HTTP traffic to services over the LAN

[ProofOfKeags]

Prerequisites

• [X] I have searched for existing issues that already suggest this feature, without success.

Describe the Feature Request

We want the ability to dial certain non-http services over the LAN. Currently we can only route HTTP traffic to various service interfaces over the LAN. Over tor, we can route traffic to any interface of any service, from anywhere. This is nice but tor is significantly slower than LAN and it is incredibly silly to route all the way over the tor network (6 nested encrypted tunnels across the globe), when you can route the data from one part of your house to another.

Why do we not allow this today?

Right now all services on the Embassy are on a single VLAN that is entirely contained to a single box. This means that when they bind processes to ports in their respective containers, it does not result in those ports being bound on the host. As such they cannot be dialed. At minimum in order for this to work, services would need to be able to request a conduit all the way to a host (essentially port forwarding at the host level rather than the overall network level).

There are a few challenges that arise from this:

1. Collisions: how do we deal with multiple applications that want the same port on the host? Generally, applications are pretty good at using unique ports, and so it is unlikely for this to happen often but it can still happen. Port space is only 65536 large, some of which are already inaccessible (22, 80, 443, 9050, 9051). Can we implement a switch in any reasonable way? I suspect not, because TCP is a pretty dumb protocol and ipaddr+port is the only way to uniquely dial a service in TCP. Further, we can't guarantee any property of the protocol that exceeds TCP.
2. Lifetime management: how long should a service be able to get port space on the host? Should it be dynamic, or should it be over the lifetime of the install?
3. This creates new security considerations as well. TCP is not on its own an encrypted or authenticated protocol, so any port binding that makes it all the way to the host is new surface area that is exposed.

Describe the Use Case

The specific situation that caused this to arise is that a customer wanted to be able to add Embassy hosted instances of bitcoind as peers to other instances of bitcoind that they have on other Embassies. While the customer was only considering bitcoind in this inquiry, the problem extends to every P2P application that does not conduct its P2P traffic using HTTP.

Describe Preferred Solution

I have no preferred solution at this time. But here is a possibility:

Add manifest key to the network configuration that requests a port binding to the host. Ensure that the port binding is part of the argument set that is given to the container at launch time. Collisions result in error on install.

Describe Alternatives

There are many alternatives most likely but they are better reserved for the discussion thread.

Anything else?

CC @kn0wmad

[chrisguida]

How does Tor accomplish this, and could avahi be augmented in some way to incorporate what Tor does? I assume it involves the SOCKS5 proxy

[ProofOfKeags]

Tor accomplishes this by putting everything behind the tor socks5 reverse proxy. We could try implementing it like that, we'd need to use socks5h so that host resolution was deferred to the proxy. This could be quite the undertaking though. I'm presently unaware of any off-the-shelf solution that will do this for us, but I haven't searched extensively.

[kn0wmad]

Could this method be used arbitrarily to solve this issue?

https://medium.datadriveninvestor.com/how-to-transparently-use-a-proxy-with-any-application-docker-using-iptables-and-redsocks-b8301ddc4e1e

[ProofOfKeags]

This seems to only apply to outgoing traffic.

[dr-bonez]

socks5 does not affect incoming traffic. tor maintains its own routing table for hidden services to clearnet ips and ports, so if tcp connections come in over tor it knows where to route it. I can't think of any way to apply this to clearnet connections.

[ProofOfKeags]

Right but tor somehow accomplishes a mapping from asdf...asdf.onion to a specific service. That technology does work somehow. What would it take to replicate that?

[dr-bonez]

would require socks5 on the client device

[chrisguida]

Shower thought: augment the Tor daemon to do this translation for .local as well as .onion. Since everyone is running a Tor daemon in their client device anyway, it won't be a huge jump to run this too.

Also there's a Rust Tor daemon:

https://blog.torproject.org/announcing-arti/

[chrisguida]

Can we go ahead and implement port forwarding? We need some way of accessing non-http LAN services over clearnet

@dr-bonez

[dr-bonez]

This is slated for 0.3.4

[chrisguida]

temporary workaround: ssh tunnel

ssh -L 127.0.0.1:50001:<package-id>.embassy:50001 start9@embassy-<product-id>.local

h/t @gStart9

[chrisguida]

Workaround for forwarding ports via LAN (not just a single client device that has SSH installed, so, for instance, Android devices):

socat tcp-l:3002,fork,reuseaddr tcp:c-lightning.embassy:3001

(Run from embassy host)

[ghost]

Thanks chrisguida for the socat tip. I wanted to make this persistent so it can survive reboots and ended up with the following should anybody want to do the same.

first install socat sudo apt install socat

make socat survive reboots sudo cp /usr/bin/socat /media/embassy/embassyfs/current/usr/bin/

create systemd service for socat in /lib/systemd/system/socat.service

[Unit]
Description=socat 8333 forward

[Service]
ExecStart=/usr/bin/socat tcp-l:8333,fork,reuseaddr tcp:bitcoind.embassy:8333

[Install]
WantedBy=multi-user.target

make socat.service persistent cp /lib/systemd/system/socat.service /media/embassy/embassyfs/current/lib/systemd/system/

make symlink for service ln -s /lib/systemd/system/socat.service /media/embassy/embassyfs/current/etc/systemd/system/multi-user.target.wants/socat.service

reboot embassy through web ui

enable systemd service sudo systemctl enable socat.service sudo systemctl start socat.service

Now I can port forward and allow bitcoind on the embassy to accept public incoming connections.

Pretty hacky but it works. Looking forward to this being built in after v0.3.4.

[chrisguida]

@jimmysatstacker Looks cool, thanks! For the record, this will not survive past OS updates, since /current is replaced by the new OS fs upon update.