apache
[Bug] Downgrade issue
Search before asking
- [X] I searched in the issues and found nothing similar.
Version
I'm trying to downgrade from 3.0.2 to 2.10.4
Minimal reproduce step
I'm using Kubernetes and I've deployed Pulsar with Helm. Pulsar works fine with 3.0.2 but I want to try the procedure to downgrade to 2.10.4 (in case of emergency).
What did you expect to see?
I expect that Pulsar downgrade works
What did you see instead?
Bookkeepers cannot start and from the logs I see: java.io.IOException: Error open RocksDB database Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in /pulsar/data/bookkeeper/ledgers/current/ledgers/000006.sst offset 929 size 32
Anything else?
Zookeeper and brokers seem to run correctly
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Bookkeepers cannot start and from the logs I see: java.io.IOException: Error open RocksDB database Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in /pulsar/data/bookkeeper/ledgers/current/ledgers/000006.sst offset 929 size 32
Good question. This might be something to ask on [email protected] mailing list. Pulsar 2.10.4 uses Bookkeeper 4.14.7 / RocksDB 6.10.2 and Pulsar 3.0.2 uses Bookkeeper 4.16.3 / RocksDB 7.9.2.
https://github.com/facebook/rocksdb/wiki/RocksDB-Compatibility-Between-Different-Releases
Backward compatible: newer version of RocksDB should be able to open DBs generated by all previous releases for normal configuration.
It seems that RocksDB doesn't provide out-of-the-box tooling to migrate from newer version to the older one. There would have to be a tool to export the data from the newer DB and import it with older DB version. I'm not aware of such tool for Bookkeeper.
It seems to be related to this issue: https://github.com/apache/bookkeeper/issues/3734
As far as I understand it should be fixed in Bookkeeper 4.14.8: https://github.com/apache/bookkeeper/pull/3947
So if I am right, I can't rollback from 3.0.2 to 2.10.4...
It seems to be related to this issue: apache/bookkeeper#3734
As far as I understand it should be fixed in Bookkeeper 4.14.8: apache/bookkeeper#3947
So if I am right, I can't rollback from 3.0.2 to 2.10.4...
Yes, that's right. Thanks for pointing that out and sharing the issue and PR links. The newest 2.10.x is 2.10.5, and it doesn't yet include the change. The upgrade to 4.14.8 / RocksDB 6.29.4.1 has been done in branch-2.10 (#21148), but there's no release. It would be helpful to bring this issue up for discussion in the [email protected] mailing list thread https://lists.apache.org/thread/n5zfhohvk7olodfx2gksxk9n5f2pt3h8 . This could be a justification for 2.10.6 release.
Is it possible to run all the components except for the bookies with image apachepulsar/pulsar-all:3.0.2 and the bookies with image apache/bookkeeper:4.14.8 through the Helm chart? Maybe I’m just saying a silly thing.
Is it possible to run all the components except for the bookies with image apachepulsar/pulsar-all:3.0.2 and the bookies with image apache/bookkeeper:4.14.8 through the Helm chart? Maybe I’m just saying a silly thing.
the apache/bookkeeper:4.14.8 image isn't compatible with the Helm chart. It might also be unusable since it wasn't until recently that I fixed the Dockerfiles in apache/bookkeeper.
It's better to create a custom docker image based on apache/pulsar's branch-2.10 instead (since 2.10.6 isn't available).
I just created the custom image. I will try it next week and let you know if it works correctly. Thank you
Hi @MonicaMagoniCom @lhotari Even if we ensure that the bookies are still on BK 4.14, at some points user will have to perform the upgrade to newer Pulsar/BK versions.
IIUC the write format used by BK 4.15, is not compatible with BK 4.14 (I'm referring to server versions). I don't think this is acceptable neither for Pulsar and BookKeeper.
I believe @hangc0276 found the proper way to upgrade/downgrade without issues in https://github.com/apache/bookkeeper/pull/3793
From my understanding it should be done like this:
- before upgrading to 3.0, set ´dbStorage_rocksDB_checksum_type=kCRC32c´ in the bookies
- upgrade bookies -> now the bookies should continue writing in format
kCRC32c, avoiding the new defaultkXXH3 - downgrade bookies should be fine since the format didn't change
But at the same time, BK uses two different formats for different purposes
- ledger index -> kCRC32c
- entry log location-> kxxHash
but the new setting would set both. Since they're different that would end up in a breaking change in one of the two. So there's no really way at the moment to keep full compatibility
@hangc0276 Am I correct?
