onnxruntime_backend
onnxruntime_backend copied to clipboard
Add support for sharing an ORT session
For every instance in a model instance group a new ORT session is created. This code adds support to share a session per instance group. This support can be enabled by defining 'share_session' to true in triton model config "parameters". Example: parameters [ ..... { key: "share_session" value: {string_value: "true"} } ]
This is a global parameter and cannot be defined per instance group. The user should determine if the parameter makes sense for their setup. GetInstanceGroupName function is added to find the instance group name through regex search over instance name.