HDDS-10341. Implement a task/framework in Recon to sync full/delta SCM Metadata DB updates at regular intervals.

[devmadhuu]

What changes were proposed in this pull request?

This PR adds task and framework in Recon to do full SCM metadata DB sync or delta updates at regular intervals.

For Recon to have an accurate and updated information related to various states of containers, blocks pending for deletion or to process any other SCM metadata information in Recon, it is important to have full SCM DB snapshot sync at startup of Recon or fall back on full SCM DB snapshot in failure scenarios like while applying delta updates from SCM metadata DB.

There are some gaps where Recon may not know of updates in SCM metadata. E.g. Recon may be down and during downtime, some containers may get created and deleted in SCM, so Recon will never get to know about such containers even when it is started again.

If we implement delta updates and during Recon downtime, this sync task will try to sync SCM DB delta updates since its rocks DB last sequence number and apply all delta updates to various SCM tasks in Recon.

This PR also re-initialize SequenceIdGenerator, ReconPipelineManager and ReconContainerManager in-memory data of pipelines, containers on full scm DB snapshot and keeps their in-memory data upto date with incremental/delta updates of SCM metadata DB in Recon.

Another change is done to separate out NODES table from SCM snapshot DB as with every new node registration or updating the node state, ReconNodeManager was changing the sequence number of Recon SCM snapshot DB which was creating problem when Recon ask for delta updates with increased sequence number of same snapshot DB.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10341

How was this patch tested?

Ran existing and new Junit based integration tests.