Clientside passed Callbackhandler not triggered when streaming on LangserveRunnable

[DJ2695]

Me and some colleagues observed an issue, where Langchain Callback Handlers get not triggered if passed as callbacks on Clientside within the Streaming function. The Runnable contains RAG Components like a Retriever. The chain/runnable was tested without Langserve and the callbacks were triggered (on_retriever_end, on_new_llm_token, on_chat_model_start) just as expected. Using Langserve's RemoteRunnable, this behaviour changed.

Are we missing out on some important information regarding clientside streaming on a RemoteRunnable or could this be an implementation Issue?

The following code shows, how it looks like

def azure_ai_chat():
    return RemoteRunnable("http://localhost:8001/azure-ai-search/")

# Langserve Runnable: Testes and Works as endpoint as well as without Langserve
runnable = azure_ai_chat()  # type: Runnable

ai_response = cl.Message(content="", elements=[], actions=[])
token_usage = TokenUsage()
async for chunk in runnable.astream(
    {"question": message.content},
    config=RunnableConfig(callbacks=[cl.LangchainCallbackHandler(),
                                     TokenUsageCallbackHandler(
                                         "gpt-4", token_usage),
                                     DocsMessageHandler(ai_response)
                                     ]),

):
    # Stream specific code
    if "answer" in chunk.keys():
                await response_msg.stream_token(chunk["answer"].content)
                
    if chunk and "context" in chunk.keys():
        relevant_docs = chunk["context"]