model_server
model_server copied to clipboard
Remote tensor enablement via C-API
- Enable usage of OpenCL buffers via OVMS C-API
- Enable usage of VA surfaces via OVMS C-API
- Enable setting output buffers for C-API inference to avoid copy
- Add C-API Async inference
Sync does work only with models. The same is true for setting output & remote tensors.
TODO:
- async error handling
- async & model reload/unload compatibility
- add negative tests
- add tests for models with multiple inputs/outputs
- TBD dispose getting context out of loaded model
🛠 Summary
JIRA/Issue if applicable. Describe the changes.
🧪 Checklist
- [ x] Unit tests added.
- [ ] The documentation updated.
- [ ] Change follows security best practices.
Can you include in tests swapping the model version while running the inferences?
Can you include in tests swapping the model version while running the inferences?
Async compatibility with online modification is not included in this MR as mentioned in description.