python_backend
python_backend copied to clipboard
Implement GetBackendAttribute to provide instance kind hint
core: https://github.com/triton-inference-server/core/pull/109 example: if model has below config and build with ENABLE_GPU=OFF:
name: "add_sub"
backend: "python"
input [
...
]
output [
...
]
# not providing instance group
#instance_group [{ kind: KIND_CPU }]
Before this change, Triton will always fill KIND_GPU on GPU system and result in:
...
I0816 00:00:09.814568 21489 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: add_sub (GPU device 0)
E0816 00:00:09.814760 21489 model_lifecycle.cc:596] failed to load 'add_sub' version 1: Internal: GPU instances not supported
...
After:
...
I0815 23:57:16.036978 21355 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: add_sub (CPU device 0)
I0815 23:57:16.209357 21355 model_lifecycle.cc:693] successfully loaded 'add_sub' version 1
...
And config API will return:
... "instance_group":[{"name":"add_sub","kind":"KIND_CPU","count":1,"gpus":[],"secondary_devices":[],"profile":[],"passive":false,"host_policy":""} ...