Electrum wallets tx history

[martonp]

Implement WalletHistorian for btc/clone electrum wallets.

[ukane-philemon]

So I took a nose dive to investigate the possibility of having this on the app and I'd like to drop my findings here for input and suggestions.

We can get the address history of an electrum wallet using the getaddresshistory method (returns an array of {txID, blockHash}), and then retrieve the verbose info for each transaction from blockchain.transaction.get.

But there are a few issues.

1. Searching for a single wallet tx: We can easily get the tx info from the blockchain.transaction.get but we need to 1. Verify that the wallet owns that tx: The aforementioned electrum method returns SigScript and Witness fields in the tx input info, both are not provided at the same time, so we need a fool-proof way to detect that the wallet authored the tx by checking the input or was part of the output (can check ownership of the output address). 2. The block number is not part of the tx info, just the hash: A quick but maybe less efficient way is to fetch all txs for all wallet addresses and find the tx there, then use the block number tied to it, Or find a service that's free to use and request for the block number using the block hash, Or cache block numbers for all wallet tx in our end for electrum wallets.

2. Fetching all txs for a wallet:

   1. We need to be able to detect the type of tx: send or receive. @buck mentioned it's possible to get the input address from the last push of the Witness, the same apply to SigScript?
   2. Tx Amount: We've got the output amounts but not all might belong to the wallet (depending on if it's a send or receive), is it better to just show the output details on the FE? If we can figure out the tx type, we should be able to get the correct amount.
   3. Fees: Fees paid are not part of the tx info, cool we don't have this info for some wallets?

Getting the txs for an electrum wallet looks like this:

1. Get all addresses using listaddresses
2. Fetch wallet txs for all addresses using getaddresshistory
3. Fetch tx info for each tx.

Most of the methods listed here have already been implemented except: listaddresses.

Let me know if I missed anything.

[martonp]

Thanks for doing the research. It seems like there will be many challenges implementing this without the go electrum wallet that @dev-warrior777 is working on.

@dev-warrior777 , it shouldn't be too hard to add a ListSinceBlock method to the go-electrum-client library, right?

[ukane-philemon]

It seems like there will be many challenges implementing this without the go electrum wallet that @dev-warrior777 is working on.

@dev-warrior777, is this still in the works?

[dev-warrior777]

This is ongoing. Please consider as a WIP and I will try to keep my progress doc up to date.

go-electrum-client - goele for short - already implements an ElectrumX server pass-thru to do getaddresshistory on it's current API for any address.

The wallet itself keeps transactions that pay to one of it's own addresses which can accessed with GetWalletTx. And ValidateAddress can tell you if the address input is both valid and a wallet address. Is this enough?

Two caveats:

1. blockchain.transaction.get has 2 forms - just the raw tx bytes - and a verbose version that sends back a lot of info in addition. Sadly several servers do not implement the verbose version at all and I try not to use it much.
2. Electrum servers throttle how many requests you can make over time so you probably could not do very many requests like this in a short time.

https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html

[ukane-philemon]

blockchain.transaction.get has 2 forms - just the raw tx bytes - and a verbose version that sends back a lot of info in addition. Sadly several servers do not implement the verbose version at all and I try not to use it much

Are the raw tx bytes the same as the verbose resp? I doubt.

Electrum servers throttle how many requests you can make over time so you probably could not do very many requests like this in a short time.

Do you know what the rate limit could be?

[dev-warrior777]

Are the raw tx bytes the same as the verbose resp? I doubt.

exactly the same but you only get that one field. In chapp electrum code he uses the struct from the verbose one but often fills it in (sparsely) from calling the non-verbose one.

Do you know what the rate limit could be?

It is different depending on your usage and which proto methods you are using. The server has the concept of a session and sessions have costs associated with different actions. Polling block headers will soon be throttled for instance but I have not studied deeply into the server code for anti-ddos measures it takes. When I get difficult responses from the server I read the real electrum client python in interface.py, network.py and session.py

[dev-warrior777]

Test: send n * GetAddressHistory with no delays


regtest

server on localhost
server has no other clients
wallet address has 3 txs

 10,000 iterations of GetAddressHistory - 2501 seconds
  3,000 iterations of GetAddressHistory - 74   seconds
  1,000 iterations of GetAddressHistory - 1   second

testnet

server remote
server is busy & has other clients
wallet address has 0 txs (server still returns the data result) but the payload is smaller

 10,000 iterations of GetAddressHistory - 1   second
  3,000 iterations of GetAddressHistory - 1   second
  1,000 iterations of GetAddressHistory - < 1 second

so if an address has txs that seems to be the diff .. 

The test is not conclusive but I was surprised.

[dev-warrior777]

@dev-warrior777 , it shouldn't be too hard to add a ListSinceBlock method to the go-electrum-client library, right?

It could be implemented by directly calling GetAddressHistory on ElectrumX server - but would be very expensive.

https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html#blockchain-scripthash-get-history

When you send an address (a scripthash) you get the complete history of txids. No way to limit to a previous point in time. If the address has 1000's of transactions you get them all. Then you would want to ask for tx details for each txid in another request. And if you do for many addresses the server will throttle you for sure as it's resource usage counter will pass COST_SOFT_LIMIT

https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#resource-usage-limits

[dev-warrior777]

For the goele wallet we keep txs, utxos, stxos in the local client database so getting those is trivial. Stxos are spent utxos that point to the spending transaction and spending block.