Fee estimation APIs getExpectedWaitTime/getRecommendedFee return invalid values

[MrStahlfelge]

The APIs getExpectedWaitTime/getRecommendedFee return invalid values for some weeks on some nodes.

By the time of writing this, calling http://213.239.193.208:9053/transactions/waitTime?txSize=1000&fee=1000000 returns 10700421007 This means, a tx with min fee and of 1 KB would wait 20 years before executing.

On the other hand, requesting the recommended fee for this tx to wait only 2 minutes http://213.239.193.208:9053/transactions/getFee?txSize=1000&waitTime=2 returns 437636, a value less than the one used above.

[knizhnik]

This two heuristics are calculated in different ways, although them are intended to be related. And both of them can produce relevant results only if server is running for some time and collect enough statistics. Expected wait time is calculate based on average transaction processing time: time elapsed since sever start divided by number of processed transactions. Such formula is based on the assumption that there is constant stream of transactions which server is processing, If such expectation is wrong and ot of the time server is just waiting for input, then this formula will definitely produce wrong results and have to be changed.

Concerning proposed fee, it is based on the constructed histogram. The more transactions we have proceeded, the more procise results can be produced based on this histogram. There is "poolHistogram" API request which shows content o this hostogram. Can you try it?

[MrStahlfelge]

I can't try it because right now the server returns better-looking values. Anyway, I would prefer to let someone look at the output actually capable of interpreting the values that it returns.

I can't say how long the server was online when the request was made, maybe it was restarted short before. But even then, showing a 20 years execution time to users is unfortunate, especially when on the other hand the recommended fee for a two minute execution is even less than the selected one. Besides, I am not sure if the estimation API works as intended at all. When we had post-merge problems and execution times of 60 minutes, it was usually returning 0 execution time.

[pragmaxim]

@MrStahlfelge IMHO the problem is that the stats depend on calling ErgoMemPool#remove method to work properly. So if some TXs get stuck in mempool (not been removed for some reason), it causes a serious bias ...

I was analyzing this but this is one of those Integration issues that is not trivial to resolve. The correct solution is on the mepool side, to avoid having rotten TXs in mempool.

There are some edge cases, like if whole block is marked as Invalid due to some particular transaction, we drop the block but not the invalid TX that caused it and it would stay in pool https://github.com/ergoplatform/ergo/pull/1803

EDIT: I take this back, the stats API is called ONLY on tx being removed from mempool, so the problem is somewhere else.