ml-commons icon indicating copy to clipboard operation
ml-commons copied to clipboard

[META] Streaming Model/Agent prediction/execution APIs

Open jngz-es opened this issue 10 months ago • 2 comments

Is your feature request related to a problem? The meta issue to track the streaming feature progress.

What solution would you like?

  • [ ] Model predict rest api based on transport-reactor-netty4 plugin.
  • [ ] Agent execute rest api based on transport-reactor-netty4 plugin.
  • [ ] ML prediction streaming calling on arrow flight instead of transport.
  • [ ] Remote model streaming predict api by supporting sse (sever-sent events) connection with LLMs.
  • [ ] MLModelTool streaming run api.
  • [ ] Agent streaming execute api with streaming supported tools.
  • [ ] Agent streaming execute api with non-streaming tools.

What alternatives have you considered? A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context? Add any other context or screenshots about the feature request here.

jngz-es avatar Jan 02 '25 21:01 jngz-es