How to enable SpentTxId, SpentHeight, and SpentIndex in Blockbook API

[kvetak]

I am comparing the capabilities of Insight and Blockbook API. Despite the fact that Blockbook is not being recommended as general blockchain parsing API, I highly appreciate that Blockbook successfully parses / recognizes all kinds of addresses (including Bech32 bc1 or multisig addresses) which Insight does not. But…

When retrieving details about the transaction via API call, transaction outputs are currently missing SpentTxId, SpentHeight, and SpentIndex keys. In order to replicate, just check the following API results for the same transaction:

• Insight which DO support above-mentioned keys
• Blockbook V1 which DOES NOT support these keys
• Blockbook V2 which also DOES NOT support these keys
• and I have tried that with same results also in my local Blockbook installation

Not being GO expert, I can see some potentially relevant methods in API source base and structure placeholders. Hence, I would like to ask:

1. If SpentTxId, SpentHeight, and SpentIndex data are (or going to be) somehow accessible via Blockbook API?
2. If the answer for 1) is positive, then how to tweak/configure local Blockbook installation to make them operational.

[martinboehm]

Hi, the functionality is already there. It is not returned by default as it is fairly expensive for Blockbook to find the spent output. You can request spent outputs by adding a parameter spending=true to the API request, for example https://btc1.trezor.io/api/v2/tx/5e71e3d164f82794fcc93cbe6928746306ef5c38731abf182dd237ab6ba8d48a?spending=true

[kvetak]

I am curious how expansive it actually is comparing to Insight API that already has it.

Would you be willing and consider adding spending=true parameter also to Blockbook API endpoints associated with listing multiple transactions at once, e.g.:

• /api/v[1,2]/address/<address>?details=txs&spending=true
• /api/v[1,2]/block/<block>?details=txs&spending=true

Then it would be fairly easy to compare Blockbook and Insight on some referential queries involving addresses with tens of transactions with many outputs.

[martinboehm]

We will not add the spending option to the block or address API calls. The speed of spending queries is in the range of 100 outputs/second on a reasonable server, which is not enough for production use on addresses or blocks.

However, it would be fairly easy to create a special spending index, which could improve the performance at least 100 times if not more. Actually, a very early version of Blockbook used this index. We abandoned it later for other indexes, which serve our purposes better.

If you want to use Blockbook to build a transaction tree for research purposes, I could help you to create this index.

[kvetak]

Well, we (Brno University of Technology) are searching for general blockchain parser. Instead of writing it from scratch, we are first investigating existing alternatives that we can leverage (such as Insight or preferably Blockbook). Hence, the interest in this feature. We know that the majority of our queries would target retrieving bulks of transactions, and thus having txids associated with inputs and outputs is a necessity for us.

If you are up to suggesting / helping how to provide this functionality in the frame of Blockbook then I am all ears eagerly awaiting your response.

[martinboehm]

OK, I will prepare something for you.

[kvetak]

I will be looking forward to it! And we appreciate the help, since it is always easier for original author to integrate new feature. Here is contribution for a 🍺 with your name on it to keep you focused on this issue, Martin.

[martinboehm]

I have prepared for you a special version of Blockbook with optional spendingIndex. Take and build Blockbook from this repo/branch https://github.com/martinboehm/blockbook/tree/spendingIndex.

You have to run Blockbook with the parameter -spendingIndex and reindex the database. Blockbook will not start if you try to run it with this parameter but without data containing the spending index (and vice versa).

The effects of running Blockbook with -spendingIndex:

• all queries (for tx, address, block and xpub) return spending information, without any additional parameter (parameter &spending=true is not necessary)
• the DB is larger - the Bitcoin Testnet, which I tested, is about 20% larger. I expect the size increase for the Mainnet or other coins in the similar range
• indexing may take slightly more memory and be slightly slower (due to large DB), but nothing significant
• the performance impact for queries is negligible, all querie have similar performance as Blockbook without the spending index

We may merge the spendingIndex to the Blockbook master in the future but for now use this branch, please.

The idea behind your contribution for a beer stayed hidden to me, ale až někdy budu mít cestu do Brna, tak se stavím na korbel :-)

[kvetak]

We have deployed public Blockbook installation including spending info patch which is connected to Bitcoin mainchain. Everyone, who will read this issue, can use it but no guarantees on future availability are given. This installation is currently used (and can be used by others) as the benchmarking point to compare patched and unpatched version of Blockbook:

• demo of /address endpoint with spending
• demo of /tx endpoint with spending
• comparing /address request for patched and unpatched version

If I may have one small wish, Martin. Could you please add transaction identifier (i.e., txid attribute for an item of vin) also to transaction inputs for API /block endpoints? Because currently, they do not have it.

• demo of v2/block that lacks references for transaction inputs
• by the way, web front-end displaying transactions in the block also misses references for transaction inputs

Regarding :beer: contribution, I donated to the SatoshiLabs beer fund. Transaction on 22th August from exchange obfuscated this fact due to multiple outputs. Let’s propagate address 3QmuBaZrJNCxc5Xs7aGzZUK8RirUT8jRKf as the excellent destination for tips for Blockbook features implemented based on community requests :-)

[martinboehm]

@kvetak Unfortunately, the wish for spent txid/vout in the tx input is hard. Again, we do not have this information in the index.

[kvetak]

@martinboehm I will trade your working effort on this wish (implementing additional index into blockbook spendingTx fork) for container which includes both blockbook compilation and its deployment. Comparing to https://github.com/trezor/blockbook/issues/67, it uses supervisord (it is: a) without single line launech, thus more robust; and b) configuration of backend and deamon is outside of Dockerfile) to run multiple loosely dependent applications inside one container and solves golang and rocksdb version pinning for container environment. Another option would be to motivate you with way more 🍻

[martinboehm]

Implemented by -extendedindex option.