Harmony explorer sync catch up issue

[breezytm]

Describe the bug The database takes an extremely long time to sync and seems to always be behind due to more blocks behind created than the sync can import. At least in my case, it doesn't appear to be related to resource consumptions.

To Reproduce Steps to reproduce the behavior:

1. Download the binary version of harmony and the new DB from here
2. version v7211-v4.3.1-0-g65614950
3. run ./harmony --run=explorer --run.shard=0 --http.ip=0.0.0.0 --db_dir=./

Expected behavior I was expecting to run the command and the node being synced in 17 hours as suggested by tail -f latest/zero*.log | grep elapsed. I eventually released every other hour I needed to stop and start it again for it to keep it up.

Screenshots Hardware specs 48vCPUs, 128GB RAM, 7.9TB Disk (RAID 0, NVMes Samsung 980 Pro PCIe), 1GBps Up/Down

Environment (please complete the following information):

• OS: Ubuntu 20.04.3 LTS

Additional context I originally started syncing using harmony.conf file and I waited three weeks and the sync did not complete. It says 3days behind for 3 weeks. I also know of three other individuals going through the same troubles even on the discord.

{"level":"info","blocks":1,"txs":70,"mgas":24.575515,"elapsed":"1.286s","mgasps":19.100848769291805,"number":"20320377","hash":"0x1a5d24537be1e95eb33840efa183ae7b40ea8ac88449d5d945aacbdad257f7fd","cache":"91.63 MiB","age":"3d37m55s","caller":"/home/runner/work/harmony/harmony/harmony/core/blockchain.go:1559","time":"2021-12-13T15:59:40.796285826Z","message":"Imported new chain segment"}

[aidin3000]

I'm having the same issue as well. Seems the database is growing faster than I can sync. Even when running on high speed drives. I end up having to stop and restart the sync for it to get back on track.

[sophoah]

Hey, could you guys try the testnet binary, there were some improvement done on the sync logic that should improves it. Are you trying to sync an archival node ?

[aidin3000]

Tried the testnet binary and it syncs up fast now

[breezytm]

@sophoah so far so good. Currently using this setup with Pocket Network in mainnet.

[crabvk]

I'm facing the same issue, even with latest Mainnet Release v4.3.4. My explorer node is always ~103920 blocks behind the api.s0.t.hmny.io.
First I synced db as described here

rclone -P -L sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=8

Then run harmony with config

harmony.conf

Version = "2.5.0"
[BLSKeys]
KMSConfigFile = ""
KMSConfigSrcType = "shared"
KMSEnabled = false
KeyDir = "./.hmy/blskeys"
KeyFiles = []
MaxKeys = 10
PassEnabled = true
PassFile = ""
PassSrcType = "auto"
SavePassphrase = false
[DNSSync]
Client = true
LegacySyncing = false
Port = 6000
Server = true
ServerPort = 6000
Zone = "t.hmny.io"
[General]
DataDir = "/var/lib/harmony"
EnablePruneBeaconChain = false
IsArchival = false
IsBackup = false
IsBeaconArchival = false
IsOffline = false
NoStaking = true
NodeType = "explorer"
ShardID = 0
[HTTP]
AuthPort = 9501
Enabled = true
IP = "0.0.0.0"
Port = 9500
RosettaEnabled = false
RosettaPort = 9700
[Log]
FileName = "harmony.log"
Folder = "./latest"
RotateCount = 0
RotateMaxAge = 0
RotateSize = 100
Verbosity = 3
[Log.VerbosePrints]
Config = true
[Network]
BootNodes = ["/dnsaddr/bootstrap.t.hmny.io"]
NetworkType = "mainnet"
[P2P]
DiscConcurrency = 0
IP = "0.0.0.0"
KeyFile = "./.hmykey"
MaxConnsPerIP = 10
Port = 9000
[Pprof]
Enabled = false
Folder = "./profiles"
ListenAddr = "127.0.0.1:6060"
ProfileDebugValues = [0]
ProfileIntervals = [600]
ProfileNames = []
[RPCOpt]
DebugEnabled = false
RateLimterEnabled = true
RequestsPerSecond = 50000
[Sync]
Concurrency = 6
DiscBatch = 8
DiscHardLowCap = 6
DiscHighCap = 128
DiscSoftLowCap = 8
Downloader = true
Enabled = true
InitStreams = 8
MinPeers = 6
[TxPool]
BlacklistFile = "./.hmy/blacklist.txt"
[WS]
AuthPort = 9801
Enabled = true
IP = "0.0.0.0"
Port = 9800

Have also tried

[Sync]
Downloader = false

but it didn't make any difference. Also,

curl 127.0.0.1:5000/node-sync

returns true, which isn't correct, obviously.