i1050 indexing - explorer functionality

[jellymlg]

Added optional indexing to node witch uses a separate "extra" database for portability. It can be enabled by setting the "ergo.node.extraIndex" property to true in the config file.

New API endpoints in /blockchain :

• ​/blockchain​/transaction​/byId​/{txId} : Retrieve a transaction by its id
• ​/blockchain​/transaction​/byIndex​/{txIndex} : Retrieve a transaction by global index number
• ​/blockchain​/transaction​/byAddress​ : Retrieve transactions by their associated address
• /blockchain​/transaction​/range : Get a range of transaction ids
• ​/blockchain​/box​/byId​/{boxId} : Retrieve a box by its id
• /blockchain​/box​/byIndex​/{boxIndex} : Retrieve a box by global index number
• ​/blockchain​/box​/byAddress​ : Retrieve boxes by their associated address
• ​/blockchain​/box​/unspent​/byAddress​ : Retrieve unspent boxes by their associated address
• /blockchain​/box​/range : Get a range of box ids
• /blockchain​/box​/byErgoTree​ : Retrieve boxes by their associated ergotree
• /blockchain​/box​/unspent​/byErgoTree​ : Retrieve unspent boxes by their associated ergotree
• ​/blockchain​/token​/byId​/{tokenId} : Retrieve minting information about a token
• ​/blockchain​/balance​ : Retrieve confirmed and unconfirmed balance of an address

Also fixed a bug that caused the node to fail reading config file on Arch Linux due to encoding issue.

There are also a few minor changes due to the improvements made in database insertion.

(closes #1050)

[pragmaxim]

@jellymlg I feel quite stupid now but I was just studying the limits and it improved a lot such that most of http client/server software has much higher limits on the URL size even though the http spec defines limits. I was hitting this 5+ years ago all the time but now it seems sort of limit free.

[jellymlg]

@jellymlg I feel quite stupid now but I was just studying the limits and it improved a lot such that most of http client/server software has much higher limits on the URL size even though the http spec defines limits. I was hitting this 5+ years ago all the time but now it seems sort of limit free.

No, it actually fixed an issue i forgot even existed. When querying large addresses sometimes the result would be none and i couldn't figure out why. As it turns out, the addresses probably got truncated, and therefore obviously no result would be found. Now large addresses work properly.

[pragmaxim]

@jellymlg Ha, that's exactly what I experienced 5 years ago, good then :+1:

[jellymlg]

benchmarks module compilation fails, probably your tooling not even trying to compile it. To fix, add Seq->Array conversion in LDBStoreBench.scala, lines 40:

    val toInsert = bts.map(bt => idToBytes(bt.headerId) -> bt.bytes).toArray

and 56:

    val toInsert = bts.map(bt => idToBytes(bt.headerId) -> bt.bytes).toArray

Please add ScalaDoc for new classes and methods.

"extends BlockSection" for new entities is misleading, BlockSection is for parts of Ergo block (header, transactions, extension, state transformation proofs) .

Why do you need to put serializers under HistoryModifierSerializer also ?

fixed LDBStoreBench, added ScalaDoc, made an alias for BlockSection, separated serializers

[jellymlg]

@jellymlg how forks processing works for ExtraIndexer? Cant' find this part

I'm not processing forks, but reverting changes shouldn't be too difficult. Also, how to detect when a fork happens?

[pragmaxim]

The biggest challenge with blockchain persistence is reverting forks as one needs a versioned storage capable of rollback or persist blockchain by Event Sourcing where you can rollback to a snapshot made at each height. If it wasn't for this consensus challenge, blockchains would be wayyyyy easier to implement.

[kushti]

@jellymlg how forks processing works for ExtraIndexer? Cant' find this part

I'm not processing forks, but reverting changes shouldn't be too difficult. Also, how to detect when a fork happens?

It seems, currently .rollback methods of the wallet and state are called just. I can add a signal also, with block ids to remove, then ExtraIndexer can subscribe for it

Another option, existing Rollback command could be sent not to ErgoWalletActor just, but also ExtraIndexer

[jellymlg]

@jellymlg how forks processing works for ExtraIndexer? Cant' find this part

I'm not processing forks, but reverting changes shouldn't be too difficult. Also, how to detect when a fork happens?

It seems, currently .rollback methods of the wallet and state are called just. I can add a signal also, with block ids to remove, then ExtraIndexer can subscribe for it

Another option, existing Rollback command could be sent not to ErgoWalletActor just, but also ExtraIndexer

If you add a signal I can work with that.

[kushti]

@jellymlg done in https://github.com/ergoplatform/ergo/pull/1904, Rollback(id) signal is carrying id of a branching fullblock (header id is the same). You need to clear data after that block (you can get height of id from history if that can simplify the job)

[jellymlg]

@kushti How do I make a test for rollbacks?

[kushti]

@kushti How do I make a test for rollbacks?

I think there are two possible ways:

• actor probing tests (but currently is not sending any info to outside, so adding some monitoring info could be needed. Would be useful for getting extra indexer status in API / panel? )
• as actor has heavy state, split it into trait with state and functions, and lightweight actor interface extending the trait. Make tests for the trait (removeAfter function etc)

[jellymlg]

@kushti finished rollback with tests