facebook
MANIFEST file incompatible between 9.7.3 to 8.11.4 version
My application was initially using RocksDB 9.7.3. The MANIFEST file contains VersionEdit entries with Type (ValueType) = 26. According to dbformat.h in 9.7.3, Type = 26 corresponds to kTypeMaxValid.
Due to certain issues, we downgraded our application which was using RocksDB 8.11.4. However, during the downgrade, rocksdb::Open() failed while reading the MANIFEST file with the following error: Corruption: VersionEdit: new-file4 entry. The file storage.rocksdb/MANIFEST-728220 may be corrupted. In dbformat.h for RocksDB 8.11.4, there is no definition for Type = 26. It appears that the MANIFEST file created by 9.7.3 is not backward compatible with 8.11.4, likely when we have Type=26 VersionEdit entries.
Is there a recommended approach to open or migrate a RocksDB database created in 9.7.3 so that it can be read by 8.11.4?
This does happen only in few instances of RocksDB. Is there any option/configuration that is generating Type=26 entries.
I am attaching the manifest, manifest_dump(generated with ldb) and OPTIONS file for reference.
We don't normally promise forward compatibility, with default options, beyond about a 6 month horizon. However, this looks to be a serious bug in persisting ValueType:: kTypeMaxValid which is subject to change. That may have started with #11113 or related prior work. And incompatibility triggered with #12419 which legitimately changed the value.
Thanks for the reply. I assume by 'forward compatibility' you meant using a DB from 9.7.3 to 8.11.4 version. Please correct me if I am wrong.
One more issue we found as part of debugging this. RocksDB instance running with 8.11.4 version. The DB’s MANIFEST file has an VersionEdit entry with Type=24 (kTypeMaxValid). We opened the same DB with rocksdb version 9.7.3. Now, it generate a new MANIFEST file, but it has the same VersionEdit entry with Type=24 for the same SST file. But in 9.7.3, the Type=24 refers to kTypeValuePreferredSeqno.
Just want to confirm with you, whether this causes any unexpected behavior or corruption with wrong interpretation for Type=24 entry.
whether this causes any unexpected behavior or corruption with wrong interpretation for Type=24 entry.
It should only be used as metadata for file boundary keys and to truncate range deletions. I'll look into a fix for this.
It should only be used as metadata for file boundary keys and to truncate range deletions. I'll look into a fix for this.
Does this mean it’s safe to proceed with the 9.7.3 release even though the Type=24 entries are interpreted differently, or would you recommend waiting for the fix before upgrading to 9.7.3?
I think it should be ok to use 9.7.3. Though you won't be able to downgrade to older versions due to the issue you described.
I think it should be ok to use 9.7.3. Though you won't be able to downgrade to older versions due to the issue you described.
Though if I understand the issue correctly, a fully compacted DB or one otherwise without range tombstones would not have these problematic manifest entries.
Compacting might be too expensive(in terms of latency and CPU) to do before upgrading.
Compacting might be too expensive(in terms of latency and CPU) to do before upgrading.
@neethuhaneesha Compaction would only be needed if you want to do downgrade. Upgrading should be fine.
If you need to downgrade and compaction is not desirable, we can patch 8.11 with additional ValueType so that it can parse the MANIFEST file.
If you need to downgrade and compaction is not desirable, we can patch 8.11 with additional ValueType so that it can parse the MANIFEST file.
That would be helpful.