risingwave icon indicating copy to clipboard operation
risingwave copied to clipboard

ch-benchmark q9 data verification fails in chaos mesh test

Open lmatz opened this issue 1 year ago • 1 comments

https://buildkite.com/risingwave-test/chaos-mesh/builds/590#018dcd0c-7f3b-4bb6-92ac-719ee95386d2

The experiments made the meta pod fail for 20 seconds

lmatz avatar Feb 22 '24 04:02 lmatz

Didn't see the ERROR: cannot advance replication slot to 0/C90F020, minimum is 0/15B79B60 log

Difference between this and Q3's inconsistency (https://github.com/risingwavelabs/risingwave/issues/15057): For Q3, the row number are different, but here, both tables have 25 rows. Not sure whether they are the same root problem.

The test instance was gone, so I can't do further analysis.

fuyufjh avatar Feb 22 '24 08:02 fuyufjh

memtable spill is enabled: https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?from=1708542494000&orgId=1&to=1708547688000&var-data_source=PE59595AED52CF917&var-namespace=longcmkf-20240221-190330&var-pod=benchmark-risingwave-compute-c-0&var-search=mem_table_spill

I think it is similar to https://github.com/risingwavelabs/risingwave/issues/15057#issuecomment-1961343919. It is already fixed in #15232.

hzxa21 avatar Mar 06 '24 09:03 hzxa21