Replace hashlib

[achow101]

hashlib depends on openssl which both increases the size of our pyinstaller build, and adds additional reproducibility headaches. It would be nice if we could drop hashlib for something that is lighter. Preferably something that is pure python or a cython module.

The hash functions we use are:

• sha256
• sha512
• pbdkf_hmac
• ripemd160
• blake2s (trezor firmware stuff, not yet merged)

Trying to get rid of hashlib could be a problem as a ton of things use it so any dependencies would not have to use hashlib, and that would be a pain to figure out. Alternatively, for the deterministic build, we could drop-in replace the hashlib module with whatever alternative implementation. That way nothing has to change, just the api needs to be the same as hashlib.

[achow101]

Openssl appears to have made it more difficult to use ripemd160 (see https://github.com/openssl/openssl/issues/16994), which means that hashlib may not have ripemd160, depending on what version of openssl is available on the system where it is being used.

[achow101]

A potential solution is to overwrite hashlib.new. See https://github.com/bitcoin/bitcoin/issues/23710#issuecomment-989051165 for how that would work.

However this would affect all uses of hashlib.new, which for downstream consumers of HWI, may be surprising, as it would affect their uses too.

[jgriffiths]

@achow101 Would you consider a PR to use wally - https://github.com/ElementsProject/libwally-core ?

While its not pure python, we currently build for most arches and its available on PyPI (https://pypi.org/project/wallycore/). Wally provides dependency-free, fast versions of the hash functions listed above, in addition it also provides a number of other HWI requirements like base58/64, script functions, bip32, bech32(+m) etc which could be migrated to if desired.

Note wally is already used in the firmware for both Jade and bitbox02. If you are interested, I'm happy to hack up a strawman PR.

[achow101]

I don't think wally would be useful in this regard. The ultimate goal is to remove openssl from python, which inherently requires removing hashlib from HWI and all of its dependencies. While HWI switching to wally would resolve that for HWI, there will still be dependencies that rely on hashlib. Furthermore, the functions that wally implements are already implemented in HWI; I don't think that wally provides us with any additional benefit.