logfire icon indicating copy to clipboard operation
logfire copied to clipboard

Published 20 hours ago • verified publisher icon pydantic

LLM qualitative evaluations and labeling

Open Luca-Blight opened this issue 9 months ago • 2 comments

Description

It would be nice to have a place in the platform for this. Another option would be to allow for integration with a partner that does provide it.

Luca-Blight avatar Feb 19 '25 20:02 Luca-Blight

Yup, we're going on this very thing, see https://github.com/pydantic/pydantic-ai/issues/915 and linked pull request.

samuelcolvin avatar Feb 19 '25 21:02 samuelcolvin

That's awesome to see!

One thing that could be an interesting feature to have, particularly for online performance, is to enable the ability for another model to be set up as evaluator versus using a human.

Luca-Blight avatar Feb 19 '25 21:02 Luca-Blight