gateway icon indicating copy to clipboard operation
gateway copied to clipboard

Allow to call a callback on streaming responses

Open unsync opened this issue 2 months ago • 0 comments

Title: Allow to call a callback on streaming responses

Description: (optional) This PR allows to call a user-defined function (null-safe) with the stream content (list of chunks)

Motivation: (optional) The motivation behind this is that i run a fork of the gateway (i needed to introduce a custom middleware) and to add tracing for the stream responses, i have a working solution (transform the stream without actually changing the format, only adding the same logic i added in this PR) but having this as a native feature of the gateway would improve the overall implementation.

The idea is actually similar to what is done with the getFromCache function.

This is a first (working) proof-of-concept, if the idea of this feature is accepted, i'm more than ready to improve the implementation if needed. Regarding the impact on performance, i think that if the function is not set the impact is non-existant, and if the function is set, then the performance has to be accepted by those implementing the function.

If accepted, it could also be a first step to have some level of callbacks/hooks for those running forks to extends the open-source gateway.

unsync avatar Apr 22 '24 09:04 unsync