Aliaksandr Kuzmik

@microsoft-github-policy-service agree company="Comet"

Hi, @loadams! I hope I fixed the issues in my unit tests (they revealed themselves only in multi-gpu env). Could you please re-run the tests?

Hi @Komal-99! We're using LiteLLMChatModel with `response_format` argument when the model supports it. For example for hallucination, the response format is ```python class HallucinationResponseFormat(pydantic.BaseModel): score: float reason: List[str] ``` In...

In an ideal world, if ` "response_format" in litellm.get_supported_openai_params("ollama/gemma2:2b")` is True, then the model supports structured outputs and the metric should work fine. But sometimes the output is misleading and...

Hi @rrfaria! That's a good suggestion, thanks, we'll look into adding this soon. We'll keep you posted!

@rrfaria to give us more user context, could you please share an example code snippet showing how you are trying to use crewAi with Opik today?

Closing this as it was shipped. See the documentation page https://www.comet.com/docs/opik/tracing/integrations/crewai/

Hi @rrfaria! Sorry for the late answer, the issue was automatically closed... Could you please open a new one with the functionalities that you are missing in the current integration?...

Hi @jkseppan for such purposes you can implement your custom metric. See https://opik.docs.buildwithfern.com/docs/opik/evaluation/metrics/custom_metric. Inside the `score` method it will call all your other metrics and then aggregate the result in...

@jverre @jkseppan I created an internal ticket, we'll try to tackle it in the nearest future. The way it will likely be defined: ```python class AggregatedMetric(BaseMetric): def __init__( self, name:...