Support new AI Chat protocol

[sinedied]

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Ai Chat protocol has been updated with a separate endpoint for streaming and slightly different object shapes. We need to update the backend/frontend to match it.

There's also an NPM package implementing the client side communication that we could use in place of existing code in the frontend.