include_callback_events=True doesn't work in stream / stream_log endpoint

[JayAhn2]

Hi!

I found that include_callback_events doesn't work in stream / stream_log.

I need to check token counts when I call stream / stream_log.

Is it intended?

Many thanks :)

[eyurtsev]

Haven't gotten around to implementing it yet. Are you relying on RemoteRunnable for handling the callbacks?

Would sending the callbacks at the end of the stream work for your use case?

[JayAhn2]

@eyurtsev

Yes. I need to see callback_events at the end of the stream :)

It is required because my service only uses stream_log endpoint, and need to store token counts on each request.

[eyurtsev]

it's end of stream vs. sending the callbacks in "real time" as they occur -- so they will be intermingled with event data. At the end shouldn't be too difficult to do.

End of stream shouldn't be too difficult to do, so i'll try to sequence that soonish

[JayAhn2]

@eyurtsev

IMHO, I guess real time callback isn't so meaningful, as callbacks are required for auditing the prompt and response for my case ;)

[eyurtsev]

Hi @JayAhn2, I haven't picked up work on this yet

Quick question -- have you confirmed that you can get token counts when streaming with the model provider that you use?

I suspect that including callback events for streaming might not actually solve your use case as you'll need to run code client side to tokenize the response and count the tokens manually (even with callbacks included)