Problem: client logic for syncing state stream is not implemented

[yihuang]

sub-issue of https://github.com/crypto-org-chain/cronos/issues/725

• support sync file streamer files through both local file system and http static file server
• parse the files and feed to local versiondb.

Open Questions

could be left to separate issues.

• current sdk builtin file streamer store separate file for each tx, are there too many small files under single directories? can linux file system handle that?
  • alternative: merge messages into single file per block, create directories based on decimal prefix of block numbers.
• file streamer should write files atomically
  • just vendor and modify the file streamer implementation.
• client sync logic:
  • with single file per block, just download the block file one by one.
  • with multiple files per block, check the end-block file existence.

[yihuang]

Components

• versiondb
  • save change sets to database.
  • support grpc query service
• versiondb streamer
  • listen to state changes and save change sets to versiondb.
• file streamer
  • listen to state changes and save change sets to local files, one file per block.
• versiondb local sync-er
  • monitor file streamer output directory, read the changesets into versiondb.
• versiondb remote sync-er
  • sync file streamer output of master node through http or any other network protocol, and save to versiondb.
• file synchronizers between remote machines ^1
  • sync files between remote machines.

Wiring

Several ways to wire things up:

1. sync semi-archive node
   • use versiondb streamer.
   • run grpc query service with versiondb.
2. async semi-archive node, (this maybe helpful if versiondb commit slow down consensus commit too much).
   • use file streamer.
   • run versiondb local sync-er.
   • run grpc query service with versiondb.
3. remote semi-archive node
   • master node use file streamer
   • remote nodes run either:
     • versiondb remote sync-er
     • or file synchronizers with versiondb local sync-er
   • remote nodes run in grpc only mode, and run grpc query service with versiondb.

[mmsqe]

Actually we still need handle block file sequence, lsyncd seems only do rsync whatever is missing.

[yihuang]

yeah, lsyncd may not be a good candidate, what we need from a file sync-er is:

• fast batch transfer for legacy files
• low latency monitoring for new files, lsyncd need a few seconds.
• may need to specify syncing starting point based on block number, lsyncd can't do this.

nfs is another alternative.

[mmsqe]

Yup, work like Git. I might just test with file watch 2 modes: sync via http -> feed db with synced block file