PygmalionAI
[Usage]: Distributed Inference Without Docker.
Your current environment
The output of `python env.py`
Not needed
How would you like to use Aphrodite?
I want to run a single model on multiple machines in a network but the issue is i can't install docker on them any way i can use it without the docker? the script of distributed inference on multi node (uses docker).
https://aphrodite.pygmalion.chat/pages/usage/distributed.html
Yes you can, it's just much more difficult without docker. Are the machines physically connected to each other? If they're not, I'd recommend not doing it - it would be slow to the point of being unusable. It's likely everything you're running will timeout before a successful all-reduce/gather.
Yes you can, it's just much more difficult without docker. Are the machines physically connected to each other? If they're not, I'd recommend not doing it - it would be slow to the point of being unusable. It's likely everything you're running will timeout before a successful all-reduce/gather.
they're on a LAN network.
can you guide? kinda confused. (tried llamacpp it's good like getting 5t/s on 70b but i need more speed.)