Issue #3005: Fix ack broken entry succeed in ensemble change unsetting

[kezhuw]

Descriptions of the changes in this PR:

Motivation

Currently, in LedgerHandle.unsetSuccessAndSendWriteRequest, LedgerHandle.sendAddSuccessCallbacks could be called by PendingAddOp.unsetSuccessAndSendWriteRequest before all pending adds evaluated. This will make entries which met ack requirement in old ensemble but have not evaluated yet succeed in new ensemble.

Changes

Check succeeded entries after unsetting all pending entries in ensemble change unsetting.

Master Issue: #3005

Further thoughts

PendingAddOp.writeComplete calls LedgerHandle.sendAddSuccessCallbacks in completed branch.

Currently, LedgerHandle.sendAddSuccessCallbacks will not be called if failed bookies overlap with all pending write ensembles. So, the code in PendingAddOp.writeComplete sounds help.

But after changed, LedgerHandle.sendAddSuccessCallbacks will be called unconditionally after all pending adds complete unsetting. So I think the snippet in PendingAddOp.writeComplete does not help anymore. Can it be dropped ? I am not that confident. Post my thoughts here for evaluation.

[kezhuw]

@fpj @sijie @ivankelly Could you please take time to evaluate this ?

[StevenLuMT]

rerun failure checks or rebase the master new code,it fix some failed checks @kezhuw

[StevenLuMT]

fix old workflow,please see #3455 for detail

[StevenLuMT]

rerun failure checks