Arize-ai
[feature request] [mastra] session and metadata support
Sorry for the delay on adding some context here, I needed to wrap my head around a few things first and get my day job done!
The main thing here is utilizing the mastra threadId to be used as a sessionId in phoenix. I was able to accomplish this with some agent method wrapping to ensure all spans will use the threadId passed in as invocation options. A little hacky so I had a short discussion with some folks at mastra and opened up a pr to add the threadId to traces! Now it will be easy to grab the threadId in the mastra span processor and add the sessionId semantic convention tag. With that alone spans will be grouped into nice agent invocation traces.
Mastra allows adding metadata via telemetry options on invocations so we can just pull from there.
Lastly we need to parse spans for user input and LLM output so in a session we get the nice conversation thread.
I am happy to take on some or all of this, maybe with a little guidance on the last note. I've started getting fairly in the weeds here so I've got some good context!
I am happy to take on some or all of this, maybe with a little guidance on the last note. I've started getting fairly in the weeds here so I've got some good context!
@tylerschoppe thanks for digging into this end-to-end. That would be amazing if you could adjust here to pick up metadata and sessions as you probably have the perfect use-case. Thank you!
Should be able to jump on the first parts of this in the next few days!
Should be able to jump on the first parts of this in the next few days!
Very exciting!
I went to move the ticket into in progress but it appears I cannot.
I've got the sessions part working, just cleaning up the code. Metadata needs a little bit of thought and maybe some discussion here but I will keep them separate so things move and the reviews are broken up.
For the metadata component, I was thinking something along the lines of piggybacking on to the telemetry metadata options that can be passed in to the agent stream/generate calls as so
await weatherAgent.generate(
'What is the weather in Tokyo?',
{
telemetry: {
metadata: {
userId: 'test-user-id',
},
},
}
);
We can then extract the metadata to OpenInference meta convention and even pull out other keys such as user id so they are contextualized.
The meta is accessible on the following traces:
- agent.prepareLLMOptions - agent.prepareLLMOptions.argument.1: {"telemetry":{"metadata":{"userId":"test-user-id"}}}
- ai.generateText.doGenerate - ai.telemetry.metadata.userId: test-user-id - metadata.userId: test-user-id
- ai.generateText - ai.telemetry.metadata.userId: test-user-id - metadata.userId: test-user-id
- agent.generate - agent.generate.argument.1: {"telemetry":{"metadata":{"userId":"test-user-id"}}}
We need to decide on if we want to add the meta to the OpenInference meta attribute on only the spans they appear on or propagate to other spans in the trace. I am leaning more towards the latter, at least in some form, so meta such as user id could end up on the root span so it is available in the Phoenix sessions view.
Let me know what you all are thinking! I'm happy to share more detail if needed. In the meantime I can get started on the implementation.
For the metadata component, I was thinking something along the lines of piggybacking on to the telemetry metadata options that can be passed in to the agent stream/generate calls as so
await weatherAgent.generate( 'What is the weather in Tokyo?', { telemetry: { metadata: { userId: 'test-user-id', }, }, } ); We can then extract the metadata to OpenInference meta convention and even pull out other keys such as user id so they are contextualized.
The meta is accessible on the following traces:
- agent.prepareLLMOptions
- agent.prepareLLMOptions.argument.1: {"telemetry":{"metadata":{"userId":"test-user-id"}}}
- ai.generateText.doGenerate
- ai.telemetry.metadata.userId: test-user-id
- metadata.userId: test-user-id
- ai.generateText
- ai.telemetry.metadata.userId: test-user-id
- metadata.userId: test-user-id
- agent.generate
- agent.generate.argument.1: {"telemetry":{"metadata":{"userId":"test-user-id"}}}
We need to decide on if we want to add the meta to the OpenInference meta attribute on only the spans they appear on or propagate to other spans in the trace. I am leaning more towards the latter, at least in some form, so meta such as user id could end up on the root span so it is available in the Phoenix sessions view.
Let me know what you all are thinking! I'm happy to share more detail if needed. In the meantime I can get started on the implementation.
@tylerschoppe I think I lean towards propagating too
@mikeldking from a UX perspective it makes things nice if you want to see all the spans associated with some metadata but it is a bit of an opinionated choice. As someone who prefers this, I don't see why you'd want to prevent propagation but it could always be added as a configuration option down the road if it becomes an issue. No need to over-engineer this right now IMO