opencompass

opencompass copied to clipboard

### Describe the feature - Batch Size - Inference time - Evaluation time ### Will you implement it? - [ ] I would like to implement this feature and create...

tonysy

[Feature] 建议增加Embedding模型评测

4

### 描述该功能 Embedding模型在知识的召回起到至关重要的作用，针对Embedding的专业评测非常有价值。 ### 是否希望自己实现该功能？ - [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

jinghai

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

KevinNuNu

### Describe the feature Hi guys, The TensorRT-LLM has been released last week. It was maintained by NVIDIA with high inference performance. Link: https://github.com/NVIDIA/TensorRT-LLM Will implement it by API calling...

starlitsky2010

[Feature] I try to use instruction tuning to fine-tune llama model and get a new alpaca model. How to use compass to evaluate the local alpaca model on MMLU and other datasets

2

### Describe the feature How to use compass to evaluate the local alpaca model on MMLU and other datasets ### Will you implement it? - [ ] I would like...

sglucas

Integrate pytorch poc python api

1

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

HIT-cwh

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...

KevinNuNu

[Feat] Support Knowledge-based Retriever

2

Thank you very much for your contributions to the community. The open-compass/opencompass project **is truly outstanding**, and I envision engaging in further research based on opencompass foundation. In this Pull...

zhangyikaii

### Describe the feature something like ```python from transformers.generation import GenerationConfig self.model.generation_config = GenerationConfig.from_pretrained(path, trust_remote_code=True) self.model.generation_config.do_sample = False ``` ### Will you implement it? - [X] I would like to...

Leymore

### Describe the feature lm-evaluation-harness supports ```acc_norm``` evaluation, which is used in [huggingface leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) ``` ARC: 25-shot, arc-challenge (acc_norm) HellaSwag: 10-shot, hellaswag (acc_norm) ``` ```acc_norm``` is calculated by the result...

kirliavc