simulator: Ensure compatability across patch releases

[ValarDragon]

Background

A very important need is being able to backport many commits, and with high confidence know that they are state compatible. We want to backport most commits, as we seek to minimize the diff between latest tested-in-prod release, and main. This reduces QA effort substantially when reviewing a new release.

We have one battery of testing -- syncing against the chain. We currently have a CI flow on every commit to latest release branch, to sync through epoch and ~1000 blocks.

But we need a stronger battery - testing on a large variety of edge cases. Thankfully, there is tooling we make that produces such a battery - the simulator. We should extend the simulator to be used to test for state compatability between patch releases. What we would do to achieve this, is run a large simulation run, and then check that the app hashes at the end are equal.

This is something I've been talking about for a couple weeks, but have not yet written up.

Design

To do this, we will need two branches to have the same simulator version, but then other commits as well. (So we will need to backport simulation changes to be the same in the two branches)

Then we should make a long simulation run, with real iavl DB, etc, and then check the resultant app hash. If theres an issue we should have tooling that can tell us the app hash at every intermediate block height & tx. (Maybe we just emit app hash after every tx & block to a file that we diff between the two? Exact long term design for improving this part will become clearer after we have an SQL backend for the simulator)

Acceptance Criteria

• [ ] We have a simulation flow that allows to test minor release state compatability on large suites of randomly generated input
• [ ] Multiple minor/patch releases to test this process with
• [ ] pure go app tests, to ensure this
• [ ] Makefile commands to run this & backtrace the error origin point