Track milestones by index

[GalRogozinski]

Description

We want to track milestones according to index. Rather than loading all transactions that have the coordinator address we want to load all the transactions that have the current index we wish to track (use Obsolete_Tag).

Motivation

1. Have smaller sets
2. Perhaps will help with (investigating) #1434

Requirements

1. In LatestMilestoneTrackerImpl#collectNewMilestones load hashes for the next milestone index you are looking for.
2. Get rid of the seenMilestoneCandidate set.

Open question

How does this affect the milestoneTracker?

[luca-moser]

The current mechanism might also explain IRI consuming a modest amount of CPU. As seen the ICC network, nodes start to consume a constant stream of CPU at around 50k milestones being published:

Here the node consumes a steady 10-16% CPU time while loading hashes belonging to the coordinator address from RocksDB.

[luca-moser]

The logic must be smarter for this to work. Consider someone setting up a new node at milestone 0 where the actual current one in the network is 1000. The new node doesn't have the milestones from 0-1000 and there is no way for it to request milestone 1 or 2 from the neighbors as it lacks the data in the db.

So if the logic is simply: "load up all transactions where the index is 1-11", the node won't find any milestone candidates and hence it won't synchronize.

What works is:

1. Rarely use the old approach to load all txs hashes from the coordinator address (like every 5 minutes and up on startup (with a delay = milestone issuance interval))
2. Now the node knows the actual latest milestone
3. Use the set of indices from newest milestone in soldification queue to newest milestone in soldification queue - 10 to query the database for new milestone candidates

[kwek20]

Yes I noticed my node with this fix didnt increase in LSMI at all, havent had time to check why. I will check your suggested solution in the PR!

[GalRogozinski]

Now that compass depth is set to 0 I think this issue can be done. This is because we will always have a chain of milestones. When a node receives a new milestone we need to check that it is above the last index and then try to solidify it...

[GalRogozinski]

A good logic can be the following:

Have 2 boot modes quick/normal:

In quick mode:

1. Try to load the latest MS
2. If it is solid and ledger state is updated then start querying the consecutive indexes from the DB until you can't find it, else solidify.
3. continue scanning for the next index every second (this is the current interval in the code, maybe we enlarge it in a different issue)

In normal mode we want to cover the following cases:

1. Maybe there are milestones that are not connected in a nice chain (happened before compass depth was set to 0). It could be that the latest MS is solid in this case but a previous one is not
2. Maybe the milestones are not consecutive (old DBs)

So what we should do in normal mode is:

1. Find latest MS
2. Start scanning by index from initialSnapshot.
3. Try to solidify any milestone on the way that is not solid
4. If one cannot find a milestone at a certain index that is below latest MS log a warning. Perhaps when we have a gossip message that can request MS by index we should utilize it.
5. After reaching latest MS continue scanning consecutively and stop when you can't find it anymore
6. continue scanning for the next index every second

[GalRogozinski]

Normal mode is equivalent to running the node with revalidate I am doing an experiment with quick mode, but it is still blocked until #1448 is completed

[GalRogozinski]

As a quick way to get this working is to add a configuration that will tell us which milestone we want to start tracking from

This way we are unblocked by #1448 and we can decouple this issue from epic #1674

[GalRogozinski]

Just to make the above clearer. The plan is:

1. upon bootstrap find the latest milestone value from the snapshot and from the configuration MILESTONE_INDEX_TO_TRACK
2. Start looking for MILESTONE_INDEX_TO_TRACK. Let's save this as index in 2 variables i and i'
3. Once the milestone is found start tracking for the milestones above it and below it. Simply do i++ and i'--
4. Repeat 3. We can stop tracking i' once we reach the last solid milestone

[GalRogozinski]

Important to note the above solution is based on the assumption that milestones always reference each other, so upon solidifying we should get them all