Vicki Boykis

Cool. Just started working on this to echo `OpenaiChatCompletionsLM` and noticed we removed the tokenizer from that class - https://github.com/EleutherAI/lm-evaluation-harness/pull/1186/files and - https://github.com/EleutherAI/lm-evaluation-harness/pull/1191 Are we calling it a different way...

Ok, so here is a fun dilemma: In starting to implement the code (https://github.com/EleutherAI/lm-evaluation-harness/compare/logprob-completions?expand=1), I found that, [due to this issue](https://github.com/EleutherAI/lm-evaluation-harness/issues/1237), I needed to update the default base model to...

Hey @gmottajr, absolutely! Does the conversation and code here so far make sense? Feel free to take the branch I pushed and linked to in my comment above and develop...

The branch is called `logprob-completions` and you can find a reference of it here: https://github.com/EleutherAI/lm-evaluation-harness/compare/logprob-completions?expand=1 Feel free to find us on the [EleutherAI Discord](https://github.com/EleutherAI/lm-evaluation-harness/compare/logprob-completions?expand=1) in the #lm-thunderdome channel, same username...

@gmottajr were you able to get started on this? Let me know if you need any help, or I'm happy to put together a sample PR for us to review.

I'm happy to help on this issue if it's something you'd still like for lm-evaluation-harness to be able to bump `datasets`. I see we include it as a [parameters when...

I'm taking a look at how some other libraries handle this: - For OpenLLM, it's [passed as an environment variable on invocation](https://github.com/bentoml/OpenLLM/tree/7edbcf8a2d6a49566018a7903c28368a03600fa4) and [here](https://github.com/bentoml/OpenLLM/blob/7edbcf8a2d6a49566018a7903c28368a03600fa4/openllm-core/src/openllm_core/utils/__init__.py#L59) - vLLM passes it as a...

I want to summarize so far for my own understanding, let me know if I'm missing or mischaracterizing anything: [Datasets](https://huggingface.co/docs/datasets/en/index) is HuggingFace's library for loading datasets for model training/tuning. Some...

In this case, then, would it make sense to do the following: **In the config:** 1. Add a new field in the task YAML files based on the inventory suggested...

I have a PR in several steps to address this, the first step is making sure the completions API works. It's here: https://github.com/EleutherAI/lm-evaluation-harness/pull/1141