ci: Create a boxed (isolated) zcashd's and the zebrad for our CI tests

[gustavovalverde]

Motivation

We'd like to to stand up our Mainnet in a box or isolated environment. This is similar to "testnet in a box", producing a fully managed P2P zcash network.

The box test is a highly controlled environment so that the diff is in zebra code, not network conditions; the current test is still useful but because it can vary widely (100% performance degradation) without a zebra code change, it should be informative but non-blocking.

Designs

We would standup several zcashd's and the zebrad under test and isolate the network for all so that our zebrad under test will only sync from the zcashd's we control. We can eventually set up something more heterogeneous or all zebrad's, but the case we're trying to isolate is a zebrad under test syncing from a network of majority zcashd's.

Creating something like a managed group of VMs in GCP, all configured with a special network device, may support this without going directly to something like k8s, which brings its own complexity but I have not tested this.

Related Work

We will probably make the config for each zebrad network to only include IPs of our boxed network peers instead of the dns seeder. And if all the nodes cannot talk to any other nodes besides the ones in our box they will not gossip uncontrollable nodes to each other.

Once we can spin up N zcashd's/full nodes and put their IPs in a zebrad config file we can try low numbers and high numbers and see if there is an inflection point between number of nodes, speed, and cloud costs.

[teor2345]

the current test is still useful but because it can vary widely (100% performance degradation) without a zebra code change, it should be informative but non-blocking.

I think this is a feature of our current tests, not a bug in those tests. We should be making Zebra work well with the current network conditions, whatever they happen to be.

I was planning on talking about the recent Zebra sync issues next week at the retrospective. I'm hoping we can work out some next steps then.

We'd like to to stand up our Mainnet in a box or isolated environment. This is similar to "testnet in a box", producing a fully managed P2P zcash network.

Can we use the ECC's "testnet in a box"? They already have one, but I'm not sure exactly how it works.

[teor2345]

If we decide to do this task, I'd like to split it up before we start work on it. We could make this ticket into a tracking issue?

This task will need either:

• a zcashd cached state, or
• changes to zebrad to get it to ignore gossipped peer addresses.

We also need to decide on a test harness. If we're using the zebrad Rust acceptance test harness, that needs a separate ticket.

Can we make these design decisions before we schedule and split this task?

We might also need some Zebra changes or bug fixes. There used to be a bug where Zebra required at least 2 nodes to sync, but I'm not sure what the minimum number of nodes is now. (I think I've seen Zebra sync with 1 node, but I'm not sure if it's reliable. Testnet seems to work well with 5-8.)

There could be other required changes that we need, I haven't thought about it in detail.

[teor2345]

This isn't a priority for the current Zebra release candidate, we can re-open it if we need it.