chifra export: Allow chifra list to go to the front of the chain

[tjayrush]

[tjayrush]

Copied from #1269

Right now chira list

scans the appearance index, but only the finalized folder of that index. This means that only events from about four hours ago are presented.

Even though the staging folder has text files in it, I can still scan them pretty quickly looking for appearances. And the next time the scanner runs, it will start where it left off last time, so it will work as expected.

This will allow the user to see transactions up to 28 blocks in the past (about five minutes)

[daoleno]

It would be beneficial if we could query the most recent unfinalized blocks. I wish this feature had a higher priority. Thanks.

[tjayrush]

With the next release, chifra export <address> will go up to the six minute mark. That is, any block more than 28 blocks old. The same code can be used to search right up to the head of the chain, but the trouble is we need some way to identify to the user that a transaction is prone to be modified if they query again.

Question for you: when you say "unfinalized" blocks, do you mean 'pending' transactions (which are not yet in a block) or just those less old than six minutes but in a block.

[daoleno]

unfinalized means ripe, unripe, staging section

.
├── blooms
├── finalized
├── map
├── maps
├── ripe
├── staging
├── ts.bin
└── unripe

[tjayrush]

Ignore maps and map (these are used for another purpose).

ts.bin is the timestamps database.

blooms and finalized consist of the consolidate index chunks and the bloom filters in front of each one. Each index chunk in finalized has a corresponding bloom filter. We search these files when doing chifra list or chifra export.

staging contains a single file with all blocks older than six minutes (28 blocks) but not yet consolidated into a chunk (we consolidate when there are 2,000,000 appearance records). As of the new version, we will be searching the stage.

ripe is only used during the scraping pass. As soon as a single 2,000 block pass is finished the 'ripe' block get added to the stage and the stage gets consolidated (if it's time). The ripe folder is not searched as it's only a temporary location for the scraped data.

unripe contains blocks that are less than six minutes (28 blocks) old. These are only used by chifra list and chifra export when the user supplies the --unripe command line option.

[tjayrush]

@daoleno Did you notice that with the v0.40.0 release chifra export now goes up to and including the staging folder?

I am going to close this, as we have another issue for getting right up to the head of the chain.