transaction notification on clients sometimes doesn't happen until 2 new blocks come in

[tzarebczan]

Had it happen to me twice recently, even before the latest version. The last time, I think it waited for 2 blocks (may have published right around the time of the first one) for it to return from the blocking call. As soon as I saw the block come in, the publish finished.

We are also seeing timeouts on lbry.tv which could be related to this.

[tzarebczan]

Better, but still an issue - 2 blocks came in before it accepted the tx. Client side? Server side? Was connected to 3.209.188.122:50001

2020-05-15 11:19:23,036 INFO     lbry.extras.daemon.daemon:680: stream_abandon () {'txid': '3fdbeb5565a6b065d235ff0aafd880d8d71efc721e761ab24ede1fa0df82644f', 'nout': 0, 'blocking': True}
2020-05-15 11:19:23,123 INFO     lbry.extras.daemon.daemon:2113: Deleted file 1995.199.16.b_full.tif
2020-05-15 11:19:32,116 INFO     lbry.wallet.ledger:153: lbc_mainnet: added BlockHeightEvent(height=764890, change=1) header blocks, final height 764890
2020-05-15 11:23:27,429 INFO     lbry.wallet.ledger:153: lbc_mainnet: added BlockHeightEvent(height=764891, change=1) header blocks, final height 764891
2020-05-15 11:23:29,619 INFO     lbry.wallet.ledger:138: (lbc_mainnet) on_transaction: address=bQMwMTB6chWPi62T6Pbxd2E1Aw6XM96kNS, height=764891, is_verified=True, tx.id=d0f65a02339b000143e43ae4d8b9be95a074e44894bc9c7f6ac1d14719fdacf2
2020-05-15 11:23:29,631 INFO     lbry.wallet.ledger:138: (lbc_mainnet) on_transaction: address=bL73LKYofSDry9HSRedz9ztPaWGTZRxynQ, height=764891, is_verified=True, tx.id=d0f65a02339b000143e43ae4d8b9be95a074e44894bc9c7f6ac1d14719fdacf2```

[tzarebczan]

This should be fixed, but it seems to have caused a higher rate of missing input/mempool tx error rate.

[tzarebczan]

This is still an issue and happens when a tx is submitted when a new block comes in - it hangs until the next block is processed.

[tzarebczan]

Pleaseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

[lyoshenka]

@shyba was looking at this recently

[lyoshenka]

this is a pretty big change. we'd need to change the mempool to use zmq, which is on the roadmap for the hub rewrite. after jack is done with his current work, this could be next on his list

[tzarebczan]

Still ran into it, but seems much harder to repro!

2021-07-22 16:22:53,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:53,207 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,209 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,200 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,201 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,201 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,211 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,211 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,212 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,496 INFO     lbry.wallet.ledger:150: lbc_mainnet: added BlockHeightEvent(height=999583, change=1) header blocks, final height 999583
2021-07-22 16:22:59,207 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:59,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1

[tzarebczan]

This still continues to cause issues on Odysee and YT sync

Some symptoms: publishes fail due to timeout, users try to reupload, causing duplicates tipping fails, causing duplicates since it's not clear/handled well YT sync doesn't record channel data since the call doesn't return within a few minutes.