Potential crash-consistency vulnerability when allow_mmap_writes is enabled

[IKACE]

Hello RocksDB developers,

We've found a potential crash-consistency vulnerability for RocksDB when allow_mmap_writes is forced to be true. We simulated a crash during the process of Append calls on PosixMmapFile using a crash-consistency bug detection tool that we are currently building. And then when we re-open the database it returns an assertion failure: "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed."

An important change worth mentioning is that we observed use_mmap_writes will be turned off in function OptimizeForLogWrite and OptimizeForManifestWrite. So we changed these two functions to instead set use_mmap_writes to be true. We do not know if RocksDB's persistent mechanisms are built upon the assumption that these two optimization functions will always be invoked and thus use_mmap_writes is always false.

Expected behavior

The database should re-open without errors.

Actual behavior

The database reports an assertion failure "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed." and aborted.

Steps to reproduce the behavior

1. Compile the following test case which create a database and insert a key

# workload.cpp

#include <cassert>
#include <iostream>
#include "rocksdb/db.h"
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <unistd.h>
#include "common.h"

using namespace std;
using namespace rocksdb;

int main(int argc, char *argv[]) {
    if (argc != 2) {
        printf("Usage: %s <workload_dir>\n", argv[0]);
        exit(1);
    }
    /* Variable declarations and some setup */
    DB* db;
    Options options;
    Status ret;
    WriteOptions write_options;
    string key, value;
    int i;

    options.create_if_missing = true;
    options.allow_mmap_writes = true;
    // Set other RocksDB-specific options as needed
    write_options.sync = true;

    /* Open the database */
    ret = DB::Open(options, argv[1], &db);
    // assert(ret.ok());
    if (!ret.ok()) {
		printf("Open failed\n");
		printf("%s\n", ret.ToString().c_str());
		exit(1);
	}

    /* Put one row into the database */
    key = string(gen_string(0, KEY_SIZE));
    value = string(gen_string(0, VALUE_SIZE));
    ret = db->Put(write_options, key, value);
    assert(ret.ok());
    /* Close the database */
    delete db;
}

2. Create an empty directory testdb
3. Use gdb to run the following test case by gdb workload testdb
4. Put a breakpoint in Append function of PosixMmapFile in gdb by br env/io_posix.cc:1147
5. Run the program until the breakpoint and quit it to simulate a crash
6. Compile and run the following minimized crash-recovery program which open a database and create an iterator, the assertion failure should be observed

# checker.cpp

#include <cassert>
#include <iostream>
#include "rocksdb/db.h"
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <unistd.h>
#include "common.h"
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

using namespace std;
using namespace rocksdb;

int main(int argc, char *argv[]) {
	if (argc != 2) {
		printf("Usage: %s <workload_dir>\n", argv[0]);
		exit(1);
	}
	/* Variable declarations and some setup */
	DB* db;
	Options options;
	Status ret;
	ReadOptions read_options;
	string key, value;
	int i;

	Iterator* it;
	char printed_messages[1000];
	int fd, pos;
	int retreived_rows = 0;
	int row_present[10] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
	char db_path[10000];

	options.create_if_missing = true;
	read_options.verify_checksums = true;

	strcpy(db_path, argv[1]);
	// strcat(db_path, "/testdb");

	ret = DB::Open(options, db_path, &db);
	// assert(ret.ok());
	if (!ret.ok()) {
		printf("Open failed\n");
		printf("%s\n", ret.ToString().c_str());
		exit(1);
	}

	it = db->NewIterator(read_options);
	// assert(it->status().ok(), "Iterator creation failed");
	if (!it->status().ok()) {
		printf("Iterator creation failed\n");
		printf("%s\n", it->status().ToString().c_str());
		exit(1);
	}

}

RocksDB version

tag: v8.10.0

Linux distribution

Ubuntu 22.04.2 LTS

Filesystem version

ext4

[jowlyzhang]

Thank you for this report. Just curious if you were able to run the reproduce steps for multiple times and each time the issue always surfaced as this assertion error? "The database reports an assertion failure "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed."

[IKACE]

Hi Yu,

Thanks for looking into this! I am able to reproduce multiple times and observe the assertion failure using the steps described above. However, when running the checker on the testdb folder, sometimes it will just return without an error, but if you run the checker multiple times you could always observe this assertion failure happening.

[IKACE]

Hi RocksDB developers,

Do we have any updates on this issue?

[jowlyzhang]

Hi RocksDB developers,

Do we have any updates on this issue?

Thanks for checking. We haven't got to check this yet. Are you able to root cause what caused the inconsistency?