onnxruntime_backend icon indicating copy to clipboard operation
onnxruntime_backend copied to clipboard

Add support for sharing an ORT session

Open quic-suppugun opened this issue 2 years ago • 0 comments

For every instance in a model instance group a new ORT session is created. This code adds support to share a session per instance group. This support can be enabled by defining 'share_session' to true in triton model config "parameters". Example: parameters [ ..... { key: "share_session" value: {string_value: "true"} } ]

This is a global parameter and cannot be defined per instance group. The user should determine if the parameter makes sense for their setup. GetInstanceGroupName function is added to find the instance group name through regex search over instance name.

quic-suppugun avatar Aug 23 '22 22:08 quic-suppugun