model_server icon indicating copy to clipboard operation
model_server copied to clipboard

Remote tensor enablement via C-API

Open atobiszei opened this issue 1 year ago • 1 comments

  • Enable usage of OpenCL buffers via OVMS C-API
  • Enable usage of VA surfaces via OVMS C-API
  • Enable setting output buffers for C-API inference to avoid copy
  • Add C-API Async inference

Sync does work only with models. The same is true for setting output & remote tensors.

TODO:

  • async error handling
  • async & model reload/unload compatibility
  • add negative tests
  • add tests for models with multiple inputs/outputs
  • TBD dispose getting context out of loaded model

🛠 Summary

JIRA/Issue if applicable. Describe the changes.

🧪 Checklist

  • [ x] Unit tests added.
  • [ ] The documentation updated.
  • [ ] Change follows security best practices.

atobiszei avatar Aug 12 '24 10:08 atobiszei

Can you include in tests swapping the model version while running the inferences?

dtrawins avatar Aug 27 '24 12:08 dtrawins

Can you include in tests swapping the model version while running the inferences?

Async compatibility with online modification is not included in this MR as mentioned in description.

atobiszei avatar Sep 03 '24 11:09 atobiszei