Changing the routing of ledgers to journal directories

[1559924775]

Descriptions of the changes in this PR: issue : #2974

BUG REPORT

Describe the bug

When the journal directory is configured to be even, the ledger allocated to each journal will be uneven. The root cause is that almost all the ledger IDs generated by the LedgerIdGenerator class are even. The number of ZK temporary order nodes is determined by the cversion of the parent node. Whenever a child node is added or deleted, it will increase by 1. The LedgerIdGenerator class uses ZK's temporary order node to generate the ledgerid, and deletes the node immediately after generation. This way of use causes almost all cversions to be even, resulting in all ledgerids to be even.

To Reproduce

Steps to reproduce the behavior:

1. Configure in bk_server.conf: jouranlDirectorie=/tmp/bk-txn1,/tmp/bk-txn2
2. production data to bookie

Expected behavior

Almost all ledgerids are assigned to directory 1

Screenshots

ledgerIdGenerator class : ZKledgerIdGenerator

Changing the routing of ledgers to journal directories can fix this bug

[dlg99]

@1559924775 I hope you are still interested in fixing this issue. My PR https://github.com/apache/bookkeeper/pull/3023 fixes similar problem with routing of requests to threads, I think the routing logic change can be reused for the journals and it is faster than md5/murmur/other hashing.

As a follow up to the discussion https://github.com/apache/bookkeeper/issues/2974#issuecomment-1028054818 the change requires some additional thinking to prevent data loss during upgrade/downgrade. I think backwards compat tests can help with this, but you will need to add tests for upgrade from old version with 2 journal/2 data disks to the new version with changed routing. The test will need to repro the case when journal replay is needed etc.

[StevenLuMT]

rerun failure checks

[StevenLuMT]

fix old workflow,please see #3455 for detail