ml-commons
ml-commons copied to clipboard
[META] Streaming Model/Agent prediction/execution APIs
Is your feature request related to a problem? The meta issue to track the streaming feature progress.
What solution would you like?
- [ ] Model predict rest api based on transport-reactor-netty4 plugin.
- [ ] Agent execute rest api based on transport-reactor-netty4 plugin.
- [ ] ML prediction streaming calling on arrow flight instead of transport.
- [ ] Remote model streaming predict api by supporting sse (sever-sent events) connection with LLMs.
- [ ] MLModelTool streaming run api.
- [ ] Agent streaming execute api with streaming supported tools.
- [ ] Agent streaming execute api with non-streaming tools.
What alternatives have you considered? A clear and concise description of any alternative solutions or features you've considered.
Do you have any additional context? Add any other context or screenshots about the feature request here.