bitcoin-sv
[DISCUSSION] Running Teranode UTXO Lookup for ARC API
Summary
ARC runs a bunch of microservices, one of them could be the UTXO lookup from Teranode, and perhaps this would need to be configured so that it maintains a history of spendTxids for a long period of time (24 hours or so).
Motivation
We've has issues with people sending transactions which attempt to spend txos which are already spent in mined transactions. Since the tx isn't in the mempool anymore this doesn't raise a mempool conflict and so SV node treats it as an orphan for a little while and eventually ignores it. This is a problem for ARC users who now have a false positive that their tx hasn't been rejected when it should have been.
Description
The idea here is that maintaining a full UTXO lookup was part of the acknowledged long term plan of ARC - partly because EF verification is a LARP if the extended data isn't verified to be accurate. Now that BEEF is a possible format for ARC api to validate it's not as big a deal but that leaves the "already spent" txo problem described above.
Additional References
I'm not sure if this is still how Teranode UTXO lookup works but the idea back in the early days of ARC was to use a hash of the extended data: txid, vout, satoshis, script to check if the utxo still existed in the set. The idea being that if the data was fake then the lookup would be on a key which didn't exist. Then you'd know it was fake data. If it returned something you'd be able to return spendTxid which would allow idempotency, and if it returned the wrong spendTxid you had confirmed a double spend attempt and could reject.
We've has issues with people sending transactions which attempt to spend txos which are already spent in mined transactions. Since the tx isn't in the mempool anymore this doesn't raise a mempool conflict and so SV node treats it as an orphan for a little while and eventually ignores it. This is a problem for ARC users who now have a false positive that their tx hasn't been rejected when it should have been.
A practical solution would be for nodes to significantly reduce the timeout period for orphaned transactions in the mempool. Retaining these transactions for up to 14 days seems harmful. Adjusting this timeout to a matter of seconds would ensure these transactions transition rapidly to a REJECTED state, allowing ARC to detect and handle them efficiently.
Moreover, SPV wallets inherently cannot detect double spends effectively, and this scenario highlights that limitation. Maintaining extensive history for UTXOs spent years prior is highly inefficient, even for miners. Thus, quickly moving transactions that attempt to double spend into a REJECTED state would provide clarity and improve performance for all participants involved.
Additionally, if adjusting node behavior proves challenging or impractical (although change seems trivial), ARC could simulate this improved handling by automatically rejecting any transaction that remains in the orphan mempool beyond a short threshold, such as five seconds.
Discussed with @oskarszoon just now -- Teranode ingest http api will be privately exposed to TAAL ARC for early testing. That api endpoint will give an immediate response (accepted or errors) which will give clear rejected responses when appropriate even during this parallel period of SV node and Teranode running.
Beyond that, the libp2p subtree messages will be listened out for, and used to confirm that Teranodes are considering the txid as valid -- this can potential introduce another status: IN_SUBTREE or similar. Oli even suggested potentially we could give that subtree a confidence score based on the associated hash.