[BOUNTY - $100] Parallelise Model Loading

[AlexCheema]

• Right now we already parallelise model downloads which works great and speeds things up a lot
• However, loading the model into memory can also be slow for large models, and right now exo will do this sequentially one node at a time
• In a similar way to model downloads, we should parallelise model loading into memory
• This should probably expose some functionality of ensure_shard in the abstract InferenceEngine, maybe something like preload_model

[aybanda]

@AlexCheema submitted PR https://github.com/exo-explore/exo/pull/211 Hope it resolves your issue

If you feel like supporting me:

https://buymeacoffee.com/aybanda

[vovw]

tried to take a turn at this #360 lmk if there are any req changes

[minhdv82]

Is this bounty still open?