KMS should automatically reconnect to Ledger devices

[jleni]

If a Ledger device is disconnected, it is only possible to reconnect by closing and restarting the KMS.

I see two alternatives:

1. The KMS takes care of reconnections.
2. The connection is maintained by signatory or even respective signatory provider. In this case, signatory-ledger-tm. When a disconnection is detected, it the provider tries to reconnect and verify that the public key has not changed. In the meantime, any sign requests will be rejected and will return an error device is not connected. Retry in progress

In my opinion, I prefer option two.

@tarcieri any feedback?

[tarcieri]

@jleni FWIW, the KMS already implements option 1, and the yubihsm crate internally handles reconnecting ala option 2, namely the signing operation in Signatory will attempt to (re)connect to the HSM, and return an error if it can't.

See the stress test at the end of this post:

https://forum.cosmos.network/t/ann-tendermint-kms-v0-0-1-preview-release-with-initial-yubihsm2-support/1218

I've tested the YubiHSM backend by removing it from one USB slot, putting it in a different one, and seeing if the signing operation picks back up after that.

Ultimately the errors are punted all the way to Tendermint: the KMS makes a "best effort" to reconnect to an HSM backend when it receives a signing request, and if it can't returns an error to Tendermint.

[jleni]

Excellent, time to add option 2 to the ledger crate then.