db_bench fill crashes with segfault 11 with level_compaction_dynamic_level_bytes

[shmuelDash]

running on rocksdb v7.0.2

db_bench command is: ./db_bench --num=620000000 --block_size=4096 --key_size=32 --value_size=1024 --stats_interval_seconds=1 --use_existing_db=False --statistics=True --histogram=True --report_bg_io_stats=True --max_background_jobs=16 --subcompactions=16 --num_multi_db=16 --compression_ratio=1 --level_compaction_dynamic_level_bytes=True --seed=115075734 --db=/media/pliops/kv/data_dir/DB_0 --threads=1 --benchmarks=filluniquerandom

very close to the end of the test (probably all objects were written) db_bench crashes with the following printout: Received signal 11 (Segmentation fault) #0 /home/pliops/workspace/rocksdb/db_bench(+0xa71d5) [0x55a7ae3861d5] ?? ??:0 #1 /home/pliops/workspace/rocksdb/db_bench(+0x842c1) [0x55a7ae3632c1] ?? ??:0 #2 /home/pliops/workspace/rocksdb/db_bench(+0x2481e2) [0x55a7ae5271e2] ?? ??:0 #3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f761f8c66db] ?? ??:0 #4 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f761df5861f] ?? ??:0

last db_bench normal printout reported 9919652000 ops done, out of 10000000000 intended (620000000 X 16 multi_db)

no errors in rocksdb log there is enough space on the RAID0 on which the test runs (uses 10TB out of 14TB available)

[anand1976]

@shmuelDash The version 7.0.2 is pretty old, and I cannot tell anything form the stack trace due to lack of symbols. Can you reproduce on the latest version with debug symbols, and fewer # of ops?

[shmuelDash]

Thanks for the fast reply. Changing the version is not an option in my case.

[mdcallag]

I just finished a round of my db_bench benchmarks doing a variety of tests, although nothing with --num_multi_db. My tests used all patch releases for 6.x, 7.x and 8.x. I don't get any crashes, although for 7.0 I used 7.0.4 which is the latest from 7.0.$x while you are using 7.0.2.

Debug symbols might help to explain the crash.