python_backend icon indicating copy to clipboard operation
python_backend copied to clipboard

Implement GetBackendAttribute to provide instance kind hint

Open GuanLuo opened this issue 2 years ago • 0 comments

core: https://github.com/triton-inference-server/core/pull/109 example: if model has below config and build with ENABLE_GPU=OFF:

name: "add_sub"
backend: "python"

input [
...
]
output [
...
]
# not providing instance group
#instance_group [{ kind: KIND_CPU }]

Before this change, Triton will always fill KIND_GPU on GPU system and result in:

...
I0816 00:00:09.814568 21489 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: add_sub (GPU device 0)
E0816 00:00:09.814760 21489 model_lifecycle.cc:596] failed to load 'add_sub' version 1: Internal: GPU instances not supported
...

After:

...
I0815 23:57:16.036978 21355 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: add_sub (CPU device 0)
I0815 23:57:16.209357 21355 model_lifecycle.cc:693] successfully loaded 'add_sub' version 1
...

And config API will return: ... "instance_group":[{"name":"add_sub","kind":"KIND_CPU","count":1,"gpus":[],"secondary_devices":[],"profile":[],"passive":false,"host_policy":""} ...

GuanLuo avatar Aug 16 '22 00:08 GuanLuo