YCSB
YCSB copied to clipboard
brianfrankcooper
INSERT FAILED when using ycsb to test cassandra.
Dear All,
I have met a problem when I try to test Cassandra with YCSB. All my steps are following the instructions of YCSB document.
The terminal command is : ./bin/ycsb.sh load cassandra-cql -P workloads/workloadb -p recordcount=3000000 -p hosts=127.0.0.1 -s -p fieldlength=250 -threads 8
But in my experiments, there always an INSERT ERROR in the log. 2021-04-24 11:57:04:722 10 sec: 297225 operations; 29719.53 current ops/sec; est completion in 1 minute [INSERT: Count=297276, Max=75647, Min=85, Avg=244.15, 90=310, 99=508, 99.9=1792, 99.99=18335] 2021-04-24 11:57:14:723 20 sec: 654516 operations; 35729.1 current ops/sec; est completion in 1 minute [INSERT: Count=357295, Max=24287, Min=94, Avg=220.18, 90=272, 99=373, 99.9=1005, 99.99=2949] 2021-04-24 11:57:24:723 30 sec: 990214 operations; 33569.8 current ops/sec; est completion in 1 minute [INSERT: Count=335688, Max=499199, Min=95, Avg=234.46, 90=272, 99=377, 99.9=1218, 99.99=4371] 2021-04-24 11:57:34:723 40 sec: 1360126 operations; 36991.2 current ops/sec; est completion in 49 seconds [INSERT: Count=369918, Max=12111, Min=94, Avg=212.88, 90=260, 99=361, 99.9=1128, 99.99=5267] 2021-04-24 11:57:44:723 50 sec: 1718781 operations; 35865.5 current ops/sec; est completion in 38 seconds [INSERT: Count=358656, Max=20031, Min=94, Avg=219.45, 90=263, 99=368, 99.9=1288, 99.99=15511] 2021-04-24 11:57:54:723 60 sec: 2073781 operations; 35500 current ops/sec; est completion in 27 seconds [INSERT: Count=355005, Max=18191, Min=95, Avg=221.74, 90=268, 99=374, 99.9=1081, 99.99=13359] 2021-04-24 11:58:04:723 70 sec: 2426734 operations; 35295.3 current ops/sec; est completion in 17 seconds [INSERT: Count=352941, Max=28815, Min=98, Avg=223.04, 90=272, 99=378, 99.9=780, 99.99=12783] 2021-04-24 11:58:14:722 80 sec: 2691424 operations; 26469 current ops/sec; est completion in 10 seconds [INSERT: Count=264645, Max=25871, Min=110, Avg=228.71, 90=278, 99=386, 99.9=823, 99.99=2255] Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 2021-04-24 11:58:24:722 90 sec: 2694103 operations; 267.9 current ops/sec; est completion in 11 seconds [CLEANUP: Count=7, Max=15, Min=1, Avg=4.71, 90=9, 99=15, 99.9=15, 99.99=15] [INSERT: Count=2684, Max=11337727, Min=246, Avg=4579.92, 90=378, 99=536, 99.9=703, 99.99=11337727] [INSERT-FAILED: Count=7, Max=11337727, Min=11329536, Avg=11333632, 90=11337727, 99=11337727, 99.9=11337727, 99.99=11337727] 2021-04-24 11:58:34:723 100 sec: 2721572 operations; 2746.9 current ops/sec; est completion in 11 seconds [CLEANUP: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [INSERT: Count=27475, Max=5115, Min=245, Avg=359.99, 90=376, 99=535, 99.9=1141, 99.99=3031] [INSERT-FAILED: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 2021-04-24 11:58:40:460 105 sec: 2730613 operations; 1575.91 current ops/sec; est completion in 11 seconds [CLEANUP: Count=1, Max=2322431, Min=2320384, Avg=2321408, 90=2322431, 99=2322431, 99.9=2322431, 99.99=2322431] [INSERT: Count=9030, Max=1561, Min=281, Avg=373.62, 90=404, 99=550, 99.9=1004, 99.99=1443] [INSERT-FAILED: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [OVERALL], RunTime(ms), 105738 [OVERALL], Throughput(ops/sec), 25824.329947606348 [TOTAL_GCS_PS_Scavenge], Count, 39 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 525 [TOTAL_GC_TIME_%PS_Scavenge], Time(%), 0.49651024229699825 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0 [TOTAL_GC_TIME%PS_MarkSweep], Time(%), 0.0 [TOTAL_GCs], Count, 39 [TOTAL_GC_TIME], Time(ms), 525 [TOTAL_GC_TIME%], Time(%), 0.49651024229699825 [CLEANUP], Operations, 8 [CLEANUP], AverageLatency(us), 290180.125 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 2322431 [CLEANUP], 95thPercentileLatency(us), 2322431 [CLEANUP], 99thPercentileLatency(us), 2322431 [INSERT], Operations, 2730613 [INSERT], AverageLatency(us), 231.059519602375 [INSERT], MinLatency(us), 85 [INSERT], MaxLatency(us), 11337727 [INSERT], 95thPercentileLatency(us), 316 [INSERT], 99thPercentileLatency(us), 398 [INSERT], Return=OK, 2730613 [INSERT], Return=ERROR, 7 [INSERT-FAILED], Operations, 7 [INSERT-FAILED], AverageLatency(us), 1.1333632E7 [INSERT-FAILED], MinLatency(us), 11329536 [INSERT-FAILED], MaxLatency(us), 11337727 [INSERT-FAILED], 95thPercentileLatency(us), 11337727 [INSERT-FAILED], 99thPercentileLatency(us), 11337727
Looks like the Cassandra instance is overloaded and can not process the INSERT load.
You can increase the retry counter, reduce the number of threads or increase the Cassandra resources for a successful load phase
I just find the reason. Please make sure you have assigned enough memory for your test. Check Cassandra's config to allow it to use all the memory int the machine.
Looks like the Cassandra instance is overloaded and can not process the INSERT load.
You can increase the retry counter, reduce the number of threads or increase the Cassandra resources for a successful load phase
Can you please let me know how to increase the retry counter
