adopting rust lang

[eukreign]

Rationale

Three main reasons for adopting rust language in the LBRY SDK:

1. Significant performance improvements.
2. Ability to share code with every other programming language; including in the browser.
3. Enthusiasm on the team to use rust.

Adoption Strategies Overview

Three independent, and parallel, paths can be taken towards this goal:

1. Change out the current pyinstaller packager used for making SDK releases for rust based PyOxidizer. This is expected to improve startup time. Currently with pyinstaller every invocation of the command line requires unzip'ing the entire SDK source before Python can parse and execute while with PyOxidizer some of this extra startup work can be eliminated.
2. Re-implement parts of the SDK wallet core code in rust and import it as modules; while also providing packages for JavaScript, Go, etc.
3. Re-implement parts of the SDK file / blob management & DHT core code in rust and import it as modules; while also providing packages for JavaScript, Go, etc.

Replacing Packager

Replacing pyinstaller with PyOxidizer should mostly be a well defined and self contained task; possibly even ripe for community contributions.

Wallet

Replacing parts of the core wallet code could follow the following logical progression of releasable phases (for Python, JavaScript and Go):

1. [optional] LBRY protobuf parser. (alternately, could require language specific bindings generated for better language ergonomics and then passing the protobuf data into the rust transaction creation code in the next phase)
   • https://github.com/stepancheg/rust-protobuf
   • https://github.com/tokio-rs/prost
2. LBRY blockchain transaction parser by forking an existing bitcoin rust project:
   • https://github.com/Congyuwang/Rusty-Bitcoin-Explorer (MIT license, relatively small code base)
   • https://github.com/rust-bitcoin/rust-bitcoin (possibly compatible license but large code base)
   • https://github.com/gcarq/rusty-blockparser (GPL)
3. Signing / verifying transactions.
4. Key & seed management.

File / Blob Management & DHT

1. [optional] Blob encryption/decryption (low priority as it is already fast)
2. Blob Manager (checking and accounting of blobs is currently very expensive, freezing SDK when it happens)
3. [optional] [raw idea, needs another issue] Fast blob reassembly from files (maybe we can stop using blobs)
4. DHT
   • Packet parsing/handling (just transport, so it doesn't become unresponsive on GIL lockups)
   • [optional] Investigate if DHT data store can use less memory

[azat11]

Fast blob reassembly from files (maybe we can stop using blobs)

This would be GREAT!