Integration tests against Epoch3.0

[zone117x]

PR https://github.com/hirosystems/stacks-blockchain-api/pull/1762 boots the stacks-node into epoch2.5 (where pox-4 activates) and refactors all pox integration tests to pox-4.

Epoch3.0 is when NakamotoBlocks come online. The script that boots up the stacks-node needs some new logic to trigger this transition, those changes will be done in the regtest-env repo https://github.com/hirosystems/stacks-regtest-env

Then the integration tests here need to start specifying and confirm epoch3.0 is activated, and test the NakamotoBlock+TenureChange ingestion.

[zone117x]

Running against an epoch3 node is pretty complicated. Duplicate efforts are on-going between the stacks-regtest and Clarinet projects. Ideally the API could use an epoch3 image from Clarinet. That is still in discussion. cc @hugocaillard

[zone117x]

Increased priority for this now that we need epoch3.0 for integration tests for new Nakamoto signer event data implemented in https://github.com/stacks-network/stacks-core/pull/4269

[smcclellan]

Aiming for mid-Feb for epoch to be ready.

[zone117x]

Update: the latest signer branch in the regtest-env repo (https://github.com/hirosystems/stacks-regtest-env/pull/33) is able to boot epoch3 when using the full docker-compose workflow.

The orchestration required is significantly more complicated than epoch2.x..

• A new "stacking" script is required to run in the background and submit txs in order to keep signer-keys registered
• A new "signing" service is required to run in the background
• A new "tx-broadcaster" script is required to run in the background in order produce Stacks blocks on an interval
• The bitcoin mining script is more complicated and runs significantly slower in order to avoid other components breaking

In order to implement integration tests in this repo so they work similar to epoch2.x, we need to:

1. Combine all the above new services and scripts into the single regtest docker image -- possible but it's going to be ugly.
2. Increase the stability of the above epoch3 booting, because it is still flaky and sometimes fails to boot or will stall out (underlying causes are either still not known or are WIP).
3. Ideally increase the bootup time, block production speed, and pox cycle speed, because these are all significantly slower now, and making the integration tests event slower would be pretty painful.

Of the above action items, only the first one is actionable right now. But it's unclear how useful that will be by itself, and also unclear if right now is a good time to divert attention away from feature development and manual testing.

---

In the meantime, we are working around this with a different development and testing workflow, which @janniks and @rafaelcr are helping with:

• Write scripts to issue different stacking workflows (extends, increases, delegation, revokes, etc), manually boot the regtest env to epoch3, run the scripts, then dump the raw events csv (e.g. event-reply export).
• Use these sets of raw events inside API for unit tests, and for the development of the signer features.