lightwalletd
lightwalletd copied to clipboard
zcash
Disappearing mempool transactions
First reported to Zebra: https://github.com/ZcashFoundation/zebra/issues/9307
Zec.rocks resolved this issue for users by rolling back our Lightwalletd versions to v0.4.17 (2fb5ccd). We were previously running Lightwalletd at e720feb.
Some change between v0.4.17 and e720feb introduced this bug.
The bug is not present on Zaino, which I realize everyone will be switching over to soon. But until that is ready, having a working Lightwalletd with recent improvements is a must. (once this is resolved, can we get a LWD release? Thank you.)
I tried manually testing both GetMempoolTx and GetMempoolStream (using zcashd and grpcurl on mainnet, just waiting until transactions appear in the mempool), using the latest master branch commit (6c400288240615ad4c59e748fd876466553c2e3f) and they both seem to be working correctly. Can you provide more information about what isn't working?
Does the problem happen only with zebrad?
Or does the problem happen only if this lightwalletd submits the transaction (the wallet issues a SendTransaction gRPC to lightwalletd, which in turn submits a sendrawtransaction RPC to zcashd) and then a GetMempoolTx or GetMempoolStream doesn't return it? I can't really imagine how that would happen.
Lightwalletd compiled from master + Zebra 2.2.0. This also happened on the Zcashd 6.1.0 endpoint, so it occurs for both.
Point the Zingo wallet app at the Lightwalletd server, submit a transaction. It disappears from the transaction list until it is confirmed, due to mempool behavior that I was only able to fix by reverting to an earlier lightwalletd version.
Do you use Signal? I'm happy to put you in a group chat with the user who reported this (DM me on the Zcash forum, same username).
More details: https://github.com/ZcashFoundation/zebra/issues/9307
Point the Zingo wallet app at the Lightwalletd server, submit a transaction. It disappears from the transaction list until it is confirmed, due to mempool behavior that I was only able to fix by reverting to an earlier lightwalletd version.
Possible candidate is 90116a7c55256e597c9a86b31ed942aebaf33699, which added this check: https://github.com/zcash/lightwalletd/blob/18ee214eab2ce44985805d5688cabba7f57bd6da/common/mempool.go#L127-L131
Is it possible that getrawtransaction is returning a height value for the transaction that is not 0 (i.e. in the mempool) but also not positive (mined)? The only thing that could occur in zcashd is -1, indicating that the transaction is conflicted (cannot be mined), but IDK what zebrad might return here.
For context, the reason that check was added was to deal with a possible race condition between block generation and getrawtransaction: https://github.com/zcash/lightwalletd/pull/495#discussion_r1715714807
me:
I'm working on a fix.
Correction, I'm still (actively) trying to figure out what the problem is.
Sorry, haven't been able to spend much time, but did try reproducing this problem today using zebrad instead of zcashd, and the GetMempoolStream seems to be working (using grpcurl).
Do you know which of these two gRPCs is being used, GetMempoolTx (the older one) or GetMempoolStream (the newer one)? Or can you point me to the source code (that's making the gRPC calls), maybe I can figure it out?
I haven't been able to reproduce any bug, but I have a theory. After v0.4.17, the mempool entries returned by GetMempoolStream now specify a height of zero instead of the height of the blockchain (the latest block). This was changed in 56fe52a11a1c1e2dad2af62e471a76b9de31d376 (part of #495).
My theory is that Zingo is ignoring return items with height set to zero. Can someone check this?
@emersonian, the release I created two days ago did not include a fix for this. However, I would still like to confirm (as mentioned in my previous comment) that this is a genuine bug in lightwalletd. I'm not actively working on this (because I don't know if it's an actual bug, and, if so, I don't know how to fix it).
I'm not sure if this clears anything up but as zingo was used to test this bug I think it's relevant. I have been informed disappearing mempool txs has happened in latest zingolib dev so I looked into it today. The truth is that the tx does not disappear. txs are sorted by height so when a tx is created it is given the mempool height (chain height+1). Due to recent changes to lwd now assigning a mempool tx with height 0, the tx ends up on the other end of the list. When it's confirmed it "re-appears" because its assigned the latest block height again. I have already fixed this sorting issue in zingolib but I need to do some more work tomorrow in tests to allow it to land.
I haven't been able to reproduce any bug, but I have a theory. After v0.4.17, the mempool entries returned by GetMempoolStream now specify a height of zero instead of the height of the blockchain (the latest block). This was changed in 56fe52a11a1c1e2dad2af62e471a76b9de31d376 (part of #495).
My theory is that Zingo is ignoring return items with height set to zero. Can someone check this?
On reading through this issue I think this is likely the cause? Sorry I hadn't seen this earlier to investigate
I'm going to close this, at least for now; feel free to re-open it if the problem is seen again.