Snow!!
I've had your exact same scenario, my solution was to run on docker-compose, because in there you can specify which GPU ids to make available to each instance
And then expose their APIs and consume with another script, it would be faster if you run the openai compatible API, however if you want to add something custom like...
> @KatIsCoding can you share docker setup? I haven't have much experiance with docker. Thanks > @humza-sami were you able to figure out how to do this? I am facing...
The most important thing about the configuration is the usage of ```yml deploy: resources: reservations: devices: - driver: nvidia device_ids: ["0"] capabilities: [gpu] ``` By specifying a `device_ids` you are...