Timothy Lim

How to submit own model to leaderboard?

1

Hi there, Will it be possible to submit our own model to the leaderboard?

Hi, I found that the prompt generated from the dataset (ex: MMLU) is not wrapped according to the model's prompt template. The performance you'll get out of the model will...

There has been a number of times, where the float type will get an unlimited amount of zeros being generated. Any idea what could be the cause? I am thinking...

Is it possible to define the prompts for KV caching up-front?

1

For a lot of use cases, there is already a pre-defined system + base prompt that is used. Can we define the KV cache for these prompts up front manually?...

It seems that the system prompt is left to be `\n` or rather blank. Inspecting UltraChat (https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k?row=5), seems that no system prompt is added to the dataset. There must be...

Are we able to structure JSON output into a single line with just one whitespace?

4

### Presentation of the new feature Output JSON without wasting tokens on whitespaces and linebreaks. Example output: `{"name": : "Tim" , "age" : 25 , "interest" : "llm" }` ###...

For structuring JSON, I find the numbers (float/integers) are problematic with a consistent pattern of repetitive zeros

1

### Describe the issue as clearly as possible: When I try to structure a JSON output to float, I will occasionally hit an error which upon inspection is caused by...

As per title.

NotImplementedError: Cannot copy out of meta tensor; no data!

2

### System Info 2.0.1+cu118, driver 535.86.05 ### Information - [X] The official example scripts - [ ] My own modified scripts ### 🐛 Describe the bug When you run on...