Distributed Training & Inference architecture

[getorca]

trafficstars

Looking at what BigScience did with petals and bloom, https://github.com/bigscience-workshop/petals, it would be cool/necessary to have a distributed architecture for training as well as inference. Very very few people will be able to afford running the model, but distributing it, even at the cost of performance would make it more accessible.

[andreaskoepf]

We are discussing several candidate models. One goal of the project is to deliver at least one model that can be realistically used on decent consumer GPUs (e.g. 12 or 24GB GPU mem). If you want to participate in the discussion please consider joining the OA dev discord: https://discord.gg/HFCPfugy

[olliestanley]

We have some capability for distributed inference in the current setup