microsoft
Custom Models cannot be used as evaluator.
Add custom model with url/v1/chat/completions may cause API error when it is used in evaluator. Here is the error msg: 2025-07-07 22:46:00 +0800 6388 execution ERROR 5/5 flow run failed, indexes: [0,1,2,3,4], exception of index 0: Execution failure in 'router_eval_2.call': (WrappedOpenAIError) OpenAI API hits NotFoundError:
How to use custom model as evaluator without internet access?
Hi, @LEOCAO6 would you mind share why you want to use custom model for evaluations? Because usually local hosted small language models don't produce good results for evaluation.
Hi Alex @a1exwang , the major reason for custom model evaluation is compliance and data sensitivity. In some FSI or Automobile use cases, data must be kept in local. Besides, local LLM like DeepSeek R1/V3 can have similar capabilities of API services from hyperscaler. Thank you!
@LEOCAO6 Thanks for the clarification of your use case. We have added this feature request to our backlog.
I believe this feature is quite important, since it gives the freedom to also choose custom models which might be internal and adhere to certain standards. I would love to see this!
