Create new stacks_node_mempool_mineable_txs prometheus metric

[CharlieC3]

This metric would be similar to the stacks_node_mempool_outstanding_txs metric, however it would only count mineable TXs, and thus would only be used when the node is running as a miner or mock miner.

This can be used to better determine blockchain stalls vs when there are no transactions to include in a new block.

[obycode]

When computing this metric, we can also track other information about the mempool transactions, such as:

• Total fees
• Total blocks required to mine them
• Totals for each dimension of block cost and size
• Next expected block(s)

[obycode]

I'm trying to think through what metrics we could get from a live-running node (e.g. a mock-miner) versus what we could get from a tool that is run periodically. For the live-running node, we probably do not want to process the full mempool, since this could cause the node to get behind when the mempool is full. We could instead generate some metrics each time a block is mock-mined, including the number of transactions included, fees, total cost values. This would be interesting information, since it would indicate if there were no transactions available to mine, and it would show what the expected metadata of the next block should be.

If we go with a separate process that just runs periodically on the live chainstate from a running node, it could mine as many blocks as it takes to empty the mempool, and give a complete picture at that moment in time.

[obycode]

The mock-miner is currently only mining 1 block when it first sees the new burn block. It should be updated to instead build a new block off of each new Stacks block that arrives.

[obycode]

With that change in place, the stacks_node_last_mined_block_transaction_count metric could be used as a proxy to see if there are any transactions available in the mempool at any given block height.

[CharlieC3]

Another idea for new metrics from a discussion I had with @obycode:

• stacks_node_last_n_blocks_bytes - The size in bytes of the last N blocks, where the amount of blocks is a property of the metric itself. e.g. stacks_node_last_n_blocks_bytes{blocks=10}, or stacks_node_last_n_blocks_bytes{blocks=25}
• stacks_node_tx_bytes_(bucket|count|sum) - Histogram of the size in bytes of transactions. e.g. distribution of TXs that are at least 1KB stacks_node_tx_bytes_bucket{le=1024}

[obycode]

The simple solution that I'd like to propose for the mineable transactions is to let the mock-miner handle this. Basically, it will attempt to mine a block and report metrics for:

1. How many transactions it includes in the next block (including setting this to 0 if there are no transactions to mine)
2. The reason that it stopped mining (ran out of transactions, miner preempted, limit reached, deadline reached)

That should give us useful information like:

• There are no mineable transactions in the mempool (0 txs; out of transactions)
• There are exactly M mineable transactions in the mempool (M txs; out of transactions)
• There are at least N mineable transactions in the mempool (N txs; miner preempted, limit reached, or deadline reached)