Qwen3-Coder icon indicating copy to clipboard operation
Qwen3-Coder copied to clipboard

求 Code 模型在 SQL benchmark 的评估代码

Open nongfang55 opened this issue 1 year ago • 2 comments

看到贵团队放出了部分评估 benchmark 的逻辑,希望参考在 Spider 和 BIRD-SQL 的评估实现。这两个 benchmark 本身在 opencompass 和 harness 都没有集成

nongfang55 avatar Sep 26 '24 13:09 nongfang55

We are currently organizing prompts for two SQL-related metrics here. https://github.com/QwenLM/Qwen2.5-Coder/tree/codeqwen1_5/evaluation/text_to_sql

As for the specific evaluation scripts, we are still working ona clean open-source version.

cyente avatar Sep 27 '24 08:09 cyente

期待 release! // waiting for release!

nongfang55 avatar Sep 28 '24 07:09 nongfang55

https://github.com/QwenLM/Qwen2.5-Coder/tree/main/qwencoder-eval/instruct/bird-spider

huybery avatar Nov 14 '24 09:11 huybery