azure.ai.evaluation SDK cannot handle `conversation` input from a csv file.

[Wix97]

• Package Name: azure.ai.evaluation
• Package Version: 1.8.0
• Operating System: Windows 11
• Python Version: 3.12.9

Describe the bug In a csv file,

To Reproduce Steps to reproduce the behavior:

pip install azure.ai.evaluation==1.8.0 Setup file data.csv:

conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hello""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"

Setup file main.py

from pathlib import Path
from azure.ai.evaluation import (
    evaluate,
    CoherenceEvaluator,
    AzureOpenAIModelConfiguration,
)

dataset_path = Path("data.csv").resolve()

model_config = AzureOpenAIModelConfiguration(
    azure_deployment="xxx",
    azure_endpoint="xxx",
)


ret = evaluate(
    data=dataset_path,
    evaluators={"coherence": CoherenceEvaluator(model_config=model_config)},
    evaluator_config={"default": {"conversation": "${data.conversation}"}},
)

Expected behavior It should output coherence score

Additional context In line 346 of the code https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py

Looks like the input conversation is a string, instead of a Dict, the previous check not in doesn't take effect, because the stringified conversation json does contain the substring 'message'.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.

[luigiw]

This is the expected behavior so far. csv files as input only support query/responese format.

[ghost]

This is really strange behavior. I only discovered this after looking at the code indicated by the exception message, maybe it would be better to add some notification in the function signature or documentation.

Do you have plan to further support conversation input in csv file in the future?