testing infra: testbed

[FintanH]

We would like to simulate test networks so that we can run them locally and so that we assert state and get some introspection, see:

• https://github.com/radicle-dev/radicle-link/issues/589
• https://github.com/radicle-dev/radicle-link/issues/593

This testbed would spin up peers that could then connect to each other. They would then interact with the API and we would inspect how the gossip and replication would resolve. A bonus would be if the testbed could also simulate NAT conditions so that when we can test NAT traversal when we get around it.

Something @alexjg mentioned was https://github.com/testground/testground which seems interesting but haven't looked into the details yet.

[alexjg]

I read through the testground documentation. It's a powerful piece of machinery for sure, but it has some downsides:

1. Test plans have to be written in Go. I'm not sure whether the application under test has to be written in Go or not, but we'll definitely be working against the existing tooling to do that. I think we would probably need to write a bunch of Rust tooling for Testground to get it working for us.
2. It has no support for simulating NATed connections, which seems like a pretty core problem for us?

On the other hand it does take care of spinning up many nodes for you, it gives you a sort of control plane for synchronizing behaviour between test nodes and a pretty rich telemetry gathering infrastructure.

It's probably worth reaching out to someone at Protocol Labs to chat through it and see whether it would fit our usecase.

[xla]

Is this overlapping/superseding #68?

[FintanH]

Ya, I think it does. I'll copy over the info from there for ease of links :)

#68:

With echt sockets and echt NAT: https://github.com/danderson/natlab

Depends on radicle-dev/infra#27

[kim]

I agree that testground is probably not what we want to start off with -- it seems to mostly be a turnkey kubernetes deployment tool.

We will still have to write code to perform some scripted actions, as otherwise we'll have an idle network. I believe it would be possible to re-use that with testground should we want to spin up a 10k nodes cluster: iiuc one would just tell it to run a particular docker container.

[kim]

#606 kickstarts a simple docker-compose network, which could form the basis for more complex topologies. Some TODOs:

• [ ] arrange for each peer to believe they're binding to a lan address, but be reachable on a wan
• [ ] ~~arrange for each peer to access the wan through a virtual NAT device (natlab)~~
• [ ] set up iptables rules approximating some NAT types
• [ ] run the actual seed binary instead of a synthetic executable
• [ ] run the daemon binary (after #576), and map its API port to the host
• [ ] arrange for the integration tests to be run in a multi-process fashion (sans assertions, probably)
• [ ] add a grafana container with a persistent config?
• [ ] figure out SD for prometheus, so we don't need the indirection through graphite-exporter

[kim]

Source of internet on configuring different NAT types using iptables:

Full Cone:

iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source "public IP"
iptables -t nat -A PREROUTING -i eth1 -j DNAT --to-destination "private IP"

Restricted Cone:

iptables -t nat POSTROUTING -o eth1 -p udp -j SNAT --to-source "public IP"
iptables -t nat PREROUTING -i eth1 -p udp -j DNAT --to-destination "private IP"
iptables -A INPUT -i eth1 -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth1 -p udp -m state --state NEW -j DROP

Port Restricted Cone:

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source "public IP"

Symmetric:

echo "1" >/proc/sys/net/ipv4/ip_forward
iptables --flush
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE --random
iptables -A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT

Obviously may not cover more exotic behaviours such as hairpinning, "neighbour ports" and such, but probably good enough as a baseline.