Issue: Evaluation runs ignore child traces

[jennmueng]

Issue you'd like to raise.

We're running evaluations on a pipeline that we've already instrumented with @traceable, when those evaluations run the trace shows up fine in the normal project view: https://smith.langchain.com/public/1480ce7d-103e-426c-b416-cef4b9c670ac/r

However, in the evaluation run's trace it's empty: https://smith.langchain.com/public/37a27aba-ea90-4184-9686-01e302b4de6f/r

Suggestion:

Have the fully instrumented pipeline we have show up in the evaluation run so it's easier to review + show token count

[hinthornw]

Ah yes the handoff back to langsmith from langchain code still is manual, and the evaluation runner currently is instrumented using langchain code - I have a ticket to fix this. Does the following work for now?

from langsmith.run_helpers import as_runnable

@traceable
def my_pipeline(....):
     return ....

def predict_result(inputs: dict):
    return as_runnable(my_pipeline).invoke(inputs)


run_on_dataset(llm_or_chain_factory=predict_result, ...)

[jennmueng]

@hinthornw nope, getting a wrap_traceable item with no children:

[peterjrichens]

I'm running into a similar issue when running the official Evaluation Quick Start example. I can see the llm child_run in the json, but its not visualised. Is this the same bug or different?

[eltonio450]

same here, 10 ish runs in a trace, but only two displayed

[hinthornw]

Closing as stale