Cluster (v3.2.0) becomes unstable when more data was ingested to a existing space

[porscheme]

Please check the FAQ documentation before raising an issue

Describe the bug (required)

• Cluster becomes unstable beyond these statistics -- vertices: 50 Million -- Edges: 3.3 Billon

• We are seeing this in our TEST cluster

I20220915 16:28:09.498340   102 NebulaSnapshotManager.cpp:67] Space 61 Part 34 start send snapshot of commitLogId 89598562 commitLogTerm 2, rate limited to 10485760, batch size is 1048576
I20220915 16:28:09.498370   102 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 34] Snapshot send failed, the leader changed?
I20220915 16:28:09.498400   102 Host.cpp:355] [Port: 9780, Space: 61, Part: 34] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed!
I20220915 16:28:09.498530    53 Host.cpp:337] [Port: 9780, Space: 61, Part: 115] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 32549073, firstLogId in wal = 32538198, lastLogId in wal = 32549073
I20220915 16:28:09.498550    66 Host.cpp:337] [Port: 9780, Space: 61, Part: 124] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 97588614, firstLogId in wal = 97582907, lastLogId in wal = 97588614

Your Environments (required)

• OS: 18.04.1-Ubuntu x86_64 GNU/Linux
• Compiler: Using docker images
• CPU: lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7551 32-Core Processor Stepping: 2 CPU MHz: 1996.300 BogoMIPS: 3992.60 Hypervisor vendor: ***** Virtualization type: full L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15
• Commit id (e.g. a3ffc7d8) Not sure how to get this

How To Reproduce(required)

Steps to reproduce the behavior:

1. Create a Nebula cluster -- graphd VM count: 3 (16 vCPU, 128 GB, 2 X 2 TB Premium SSD NVMe Disks) -- metad VM count: 3 (16 vCPU, 128 GB, 2 X 2 TB Premium SSD NVMe Disks) -- storaged VM count: 9 (16 vCPU, 128 GB, 2 X 2 TB Premium SSD NVMe Disks)
2. Create a space with INT64 Vid Type, 200 Patition_num, 2 Replica Factor
3. Initially, ingest data using spark-connector -- vertices: 50 Million -- Edges: 3.3 Billon
4. Run 'SUBMITJOB COMPACT', executed successfully
5. Verify cluster is very healthy, physical size of Data & Log on disk, they are not significant
6. Ingest more data, observe cluster becomes unstable

Expected behavior

Cluster should be stable for at least 1 billion vertices and 50 billion edges

Additional context

• We did not run BALANCE DATA
• We created massive cluster to eliminate storage and memory issues

[wey-gu]

cc @Sophie-Xie

[liwenhui-soul]

the Replica Factor is 2? we suggest the replica factor is odd number, please try again with replica factor 3 or 1

[porscheme]

the Replica Factor is 2? we suggest the replica factor is odd number, please try again with replica factor 3 or 1

Thanks for the reply. I will try with replica factor 3 and report back. Which is better Vid Type, INT64 or STRING? Also, Partition_num 200 seems okay?

[liwenhui-soul]

the Replica Factor is 2? we suggest the replica factor is odd number, please try again with replica factor 3 or 1

Thanks for the reply. I will try with replica factor 3 and report back. Which is better Vid Type, INT64 or STRING? Also, Partition_num 200 seems okay?

INT64 and STRING are both ok, I think. Partition_num 200 is ok.

[porscheme]

the Replica Factor is 2? we suggest the replica factor is odd number, please try again with replica factor 3 or 1

Thanks for the reply. I will try with replica factor 3 and report back. Which is better Vid Type, INT64 or STRING? Also, Partition_num 200 seems okay?

INT64 and STRING are both ok, I think. Partition_num 200 is ok.

After setting Replica Factor 3; cluster seems to be more stable.