gateway icon indicating copy to clipboard operation
gateway copied to clipboard

Allow to call a callback on streaming responses

Open unsync opened this issue 10 months ago • 1 comments

Title: Allow to call a callback on streaming responses

Description: (optional) This PR allows to call a user-defined function (null-safe) with the stream content (list of chunks)

Motivation: (optional) The motivation behind this is that i run a fork of the gateway (i needed to introduce a custom middleware) and to add tracing for the stream responses, i have a working solution (transform the stream without actually changing the format, only adding the same logic i added in this PR) but having this as a native feature of the gateway would improve the overall implementation.

The idea is actually similar to what is done with the getFromCache function.

This is a first (working) proof-of-concept, if the idea of this feature is accepted, i'm more than ready to improve the implementation if needed. Regarding the impact on performance, i think that if the function is not set the impact is non-existant, and if the function is set, then the performance has to be accepted by those implementing the function.

If accepted, it could also be a first step to have some level of callbacks/hooks for those running forks to extends the open-source gateway.

unsync avatar Apr 22 '24 09:04 unsync

@unsync we'll make this a part of the upcoming hooks architecture. Thanks for the suggestion, but we're pausing this PR for now. Will incorporate these suggestions in the hooks merge happening soon.

roh26it avatar Jul 05 '24 06:07 roh26it