[Nakamoto] stacks-signer needs track processed blocks responses, block requests, and block sign requests

[jferrant]

Stacks-signer currently will just process blocks as they are received and trigger signing rounds accordingly. it does not currently check if it has already processed a particular block response (received from the RPC endpoint) or block request (received from the miners) or block sign request (received from other signers).

Add persistent state to the signers and update it to no longer unnecessarily trigger sign requests/block responses/ block validation requests based on prior received information.

Depends on https://github.com/stacks-network/stacks-core/pull/4130

[hstove]

@kantai is the assumption that the signer will rely on disk (ie SQLite) for state?

[jferrant]

@kantai is the assumption that the signer will rely on disk (ie SQLite) for state?

That is kind of what I was thinking, but you can choose what you like! Voluntold!

I think we have the logic in place for what to store on the side. but we have nothing for introducing persistent state or proper garbage collection. This ticket should include both those things.

[hstove]

Ok! Sounds good. I'll definitely use SQLite.

I think we have the logic in place for what to store on the side

Did you mean to say "on the signer"?

At first thought, I can imagine two ways of doing this:

• Track event observer events and simply skip one if it's already been seen / handled. The Stacks API has basically the same functionality which I'm familiar with. We'd likely only want to store the events that are relevant to us, otherwise that'd get very large.
• Track SignerEvents and have some mechanism for identifying a specific one, and skip based on that

My hunch would be that the first mechanism is more robust, but I can imagine there would be instances where more specific / queryable information could be useful.

A different question is: why is this necessary? I'm not doubting the ticket, I'm just not sure I understand the problem fully yet.

[kantai]

Ah -- yes, sorry, SQLite was exactly what I was thinking here.

[jferrant]

So right now we just store block info in a Hashmap and it grows almost indefinitely (only being removed following a successful broadcast of the signature) This garbage collection needs to be done.

We already store I think everything we need on the side inside the stacks signer runloop struct (inside hashsets and hashmaps. Specifically blocks and txs). But this needs to be garbage collected for sure.

We may not need to track transactions (as they are already written to stackerdb every time they are created following my current open PR getting merged), but we do need the garbage collection logic to clean up transactions as they get confirmed in blocks.

I would expect that all block requests are stored and persisted between crashes so that if a miner reqeusts a block be signed and we crash before we can respond to it, we know to process it on reboot.

Alternatively, we should keep blocks validation info around for a while (between crashes) as we may get antoher request to process a block that we already confirmed with an RPC call to the stacks node that it was not viable and we could clog up our stacks node with repeated calls to validate the same invalid block we have already processed.

We may have need to persist some signature shares as well from wsts for picking up dkg where we left off.

I would see this almost as three tickets:

One for garbage collecting what we already have in place One to introduce the mechanism for storing local state in sqlite The last to start adding the items we care to store between crashes