opencompass

opencompass copied to clipboard

[Feature] MBPP数据集的CN版本 mbpp_cn 能够release一下吗 ORZ，感谢大佬

1

### Describe the feature 数据集： mbpp_cn ### Will you implement it? - [ ] I would like to implement this feature and create a PR!

SrWYG

## Motivation `api_prompts` is a list of prompts in opencompass. https://github.com/open-compass/opencompass/blob/3aeabbc427b8084ea3276991348f911794a345f6/opencompass/models/base_api.py#L351-L355 However, the prediction files confused `api_prompts` with `raw_prompt`. ## Modification **Before PR**: Here is a prediction example, `predictions/ceval-college_physics.json` ```yaml...

xu-song

Process failed

3

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

sgjohnson1981

[Bug] TurboMindModelwithChatTemplate中tm_model变量可能未初始化导致报错

1

### Prerequisite - [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help. - [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass). ### Type...

PPnorain

[Feature] _set_model_kwargs_torch_dtype in _load_model will overide the torch_dtype in the config.json of any given huggingface models by default

1

### Describe the feature https://github.com/open-compass/opencompass/blob/52eccc4f0efd3ca6f272ae19efb2d7f6cc9c9dec/opencompass/models/huggingface_above_v4_33.py#L213  In most cases people who import any huggingface model by the argument --hf-path might think the "torch_dtype" in the config.json will take effect, but...

imhmhm

运行评测结果为空[Bug]

4

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。 ### 环境 {'CUDA available': True, 'CUDA_HOME': '/usr/local/cuda', 'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0', 'GPU...

badmic

meta-llama/Meta-Llama-3-8B-Instruct evaluated results is not consistent with hugging face's official results

5

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 python -c "import opencompass.utils;import pprint;pprint.pprint(dict(opencompass.utils.collect_env()))" {'CUDA available': True, 'CUDA_HOME': '/usr/local/cuda', 'GCC':...

hzgdeerHo

[Bug] Alignbench无法使用VLLM模型评测，eval阶段卡住并报错

15

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我正在使用官方支持的任务/模型/数据集进行评估。 ### 环境 ``` opencompass 0.2.6 Ubuntu 20.04 python 3.10.14 ``` ### 重现问题 -...

IcyFeather233

[Bug] 多卡时，GPU7显存占用比其他卡多30G+

4

### 先决条件 - [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。 - [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。 ### 问题类型 我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。 ### 环境 ``` {'CUDA available': True, 'CUDA_HOME': '/usr', 'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',...

dhcode-cpp

### Describe the feature 目前看到这个多语言代码数据集在一些模型评测上有较多的提及，希望能够支持一下 https://github.com/nuprl/MultiPL-E ### Will you implement it? - [ ] I would like to implement this feature and create a PR!

linbeyoung