Slow

[iamtrask]

https://github.com/OpenMined/sonar/blob/master/notebooks/Sonar%20-%20Decentralized%20Model%20Training%20Simulation%20(local%20blockchain).ipynb

The training process is quite slow. There's lots of low hanging fruit to make it faster, and I'd like to capture the issue here. The main bottlenecks as I see them are:

1. Serialization and deserialization to IPFS (this is the big one)
2. Blockchain Smart Contract Ops
3. Homomorphic Encryption and Decryption
4. Homomorphic Operations
5. Deep Learning Algos Themselves

There are plenty of redundancies in the current code.

(1) simply happens more than is necessary, and it's usually entire python objects (instead of... say...just the weights). Furthermore, it doesn't get cached in the python clients and is instead re-serialized every time it's called (very expensive). (2) many of the smart contract ops iterate through entire lists to handle gradients and such instead of constant time lookups. (3) FV uses a server that wraps R which wraps C++... the obvious thing here is to wrap the C++ directly in Python. That should speed things up considerably. (3) FV also has native support for vectors and matrices which we ignore for the sake of simplicity. Using them should make ops much faster than wrapping at the scalar level (which is what we currently use) (4) if you can see faster ways to implement these, i'm all ears (perhaps by re-implementing paillier in BLAS). There's also a GPU implementation of YASHE that's pretty cool. (5) Since addition and multiplication by a constant are the fastest operations in HE, better mini-batching should allow for considerable speedups. (5) the use of Momentum should also be quite useful given that all the gradients are public.

... this is the most obvious low hanging fruit to me. Love to see more below as you come across it.