Congestion leads to sub-optimal mining: Why are miners not optimally using block "space" for max fees?

[314159265359879]

Describe the bug Gathering info here for visibility and easy reference.

Mempool reality, there is some congestion:

An example from yesterday: In these four blocks form yesterday there are only STX transfers mined. No contract calls. And block fullness looks very low in such cases https://obycode.github.io/stacks-stats/block/?height=132494

https://explorer.hiro.so/block/0x980cf9a29ed86e606101f7f32d44744dbf902d1ed947ee00ae4f5c3d85478ea8?chain=mainnet (132494) https://explorer.hiro.so/block/0x18ac1ba47b3054bad1f0c12329c90c73e93645d2216d9bedd863e321c33b0ef5?chain=mainnet https://explorer.hiro.so/block/0xfe170e6c7798b09d76bfc0e18a4b5a11bc8ea44b7109fc861044c7a8d35ee965?chain=mainnet https://explorer.hiro.so/block/0x81b7f40e45493968c65774cf6390791bfc6f52c90c2a562ab9dc2c7432de1ddc?chain=mainnet This was the last block (123490) with contract calls in the list, with average fees around 45 STX per tx. Does that make the miner software ignore others because they're not valuable enough to be considered? https://explorer.hiro.so/block/0x0b5f523e08efe6db36960b3fc1fd0f769a0ec246b1acf1d4b8f70edd8a004b32?chain=mainnet Then block 132195 has one contract call in it with some of those STX transfers. And it is def. also not full

Block fullness of 132495

Sometimes blocks are created with thousands of transactions rather than a hundred and then contract calls are included. Not all the miners are equally effective?

Steps To Reproduce Launch a popular (degen) project on Stacks ;-)

Expected behavior I expect miners to also mine contract calls, (at any fee rate even) when there is so much of the block budget still unused. I checked a couple of popular contracts at the time such as the swap contract on ALEX and it did have some pending calls. We see exchanges use send-many calls that are pending for a very long time too.

We also see cases where there are similar transactions with higher fees in the mempool but the ones with lower fees are getting mined first. Users coming from bitcoin find this odd, can we do a better job at filtering based on STX/vB to avoid this issue? As you see on mempool.space for bitcoin transactions?

Additional context These stats are likely also relevant, mempool transactions have been in the mempool for a long time on average. I expect there are more junk transactions in there then usual, perhaps poorly constructed (bad nonce, and insufficient balance are most commonly causing unprocessable transactions), and or it may have something to do with a forking chain.

Forking/reorg chain User reported this transaction mined (incoming 3400 STX from a central exchange) Their wallet displayed the balance as available and they went ahead and create swap transactions. But then the chain forked and the funds were no longer available and the subsequent swaps failed due to insufficient funds

Swap transactions via arkadiko.finance or app.alexlab.co and they all failed.

[314159265359879]

Discord discussion: https://discord.com/channels/621759717756370964/730814571517968436/1186424827238699029

[314159265359879]

And Jude's comments

[314159265359879]

Another transaction that was mined in another fork but now is pending again according to a user. Also send from Mexc https://explorer.hiro.so/txid/0xad3b6ce558237a48ad86b42dcb3d121821915cb5fecc23be47a2072cd946267e?chain=mainnet

Average fees for STX transfer are much higher than the 0.01 STX for this transaction so that could explain why it didnt' get processed in the now canonical fork.

[shivaenigma]

This is still happening now for last 2-3 weeks :timer_clock: and it would be to good to understand the exact reasons behind this at it will effect the long term user confidence :red_square: in the STX network

Few things about economics of POW vs Proof of Transfer:

1. About fee/coinbase ratio one must note that even in bitcoin this was close to 0 in the initial years as most transaction had a fee of 0-1 satoshi/bytes. Despite this, miners included the transactions in the block. This is because in POW the miner has purchased the mining equipment for long term and it is in his self interest that the network works wells and is used by more people and in turn increases the BTC price
2. However, in case of STX most current miners seems to have no economic interest in maintaining the health of the network. They just immediately sell their STX and make profits
3. This is the same reason why POS chains like ETH have zero block reward since the miners actually does no real work to get the block. The reward is only the total fee of tx in the block that the miner has worked to validate

About the recent congestion a simple analysis can be done by seeing this block https://explorer.hiro.so/block/0xbbc9140f3b000df2e31cc76a1d98cda0a98790f347c822cf46494f07ed0e2d57?chain=mainnet Number of transactions: 15 Fee collected(14STX): 1STX fee of each of 14 transactions Coinbase Collected: 1000STX Fee/Coinbase ratio: 0.014

It is most probable here that the miner just cherry picked all transactions in the mempool that we paying atleast 1STX fee and didn't even bother seeing/checking the other tx. This miner maybe using his processing power to doing other calculations or mining the next block.

[jcnelson]

Myself and a few others have been studying this for the past few weeks. Here's what we've found:

• We believe the miners are using the default value of 30,000 ms (30s) for the subsequent block assembly time. This is consistent with them mining merely hundreds (instead of ~10,000) STX-transfers in their blocks. The mock-miner easily creates blocks with thousands of transactions using a subsequent block assembly time of 120,000 ms (2 min), which as of today is the new default value in master and in the next release (which is imminent).
• Fee assessment for STX-transfers is erroneously underpriced by between 2.5x and 3.7x. The consequence of this is that the mempool walk logic (which picks transactions for block inclusion) overestimates the price of a contract-call and a contract-publish by the same amounts; transactions that don't pay this overestimation would be erroneously ignored. Because STX-transfers use between one and three orders of magnitude less compute resources than these latter two types of transactions, the effect of this underpricing is severe -- a median transaction fee of 0.1 STX for a STX-transfer would cause the cost of a stacking transaction to be between 600 STX (5th percentile estimate) and 2700 STX (95th percentile estimate). To be clear, a median transaction fee of 0.1 STX for a STX-transfer will mean that a contract-call or contract-publish would cost 10s to 100s of STX (again, due to the discrepancy in these transactions' relative compute costs), but the 2.5x-3.7x amplification has been fixed in master.