opencompass

opencompass copied to clipboard

UnboundLocalError: local variable 'prompt_token_num' referenced before assignment and NO OUTPUTS

2

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 ``` {'CUDA available': True, 'CUDA_HOME': '/usr/local/cuda-12.2', 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.3) 9.4.0',...

qy1026

[Feature] Add a new evaluation dataset

3

### Describe the feature We want to contribute our benchmark to OpenCompass. Here is the repo:https://github.com/IAAR-Shanghai/UHGEval. However, there is an issue: in one of the tasks within this benchmark, multiple...

wyzh0912

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

Yggdrasill7D6

[Feature] Support datasets in OpenAI/evals

2

### Describe the feature [OpenAI/evals](https://github.com/openai/evals/tree/main/evals/registry/data) contains many community-sourced tasks, with about 400-500 datasets covering various languages and fields. Compared to other open-source datasets, they have very few questions, but many...

Ezra-Yu

[Bug] I downloaded the Llama-7b model on the huggingface to the local location, and then modified the path to the file location. After running the code, the following error occurred. How can I solve it?

5

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 python ### 重现问题 - 代码/配置示例 python run.py --datasets ceval_ppl --hf-path /root/pruning/llama-7b...

Rain19981998

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

yitongYao

Leymore

[Bug] Failed to reproduce llama2-70b-base on triviaqa

5

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 其他任务正常 ### 重现问题 - 代码/配置示例 from opencompass.openicl.icl_prompt_template import PromptTemplate from opencompass.openicl.icl_retriever...

DreamMr

[Bug] RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

1

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

yuanhangtangle

[Codellama assessment score with meta prompt is half lower than codellama assessment score without meta prompt]

1

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

majestichou