Kevin M Jablonka

Right now, some of the field are a bit confusing (see the comments in your open PR). Ideally, we should ask for the same thing in the form as we...

the `results` the `benchmark.bench returns after restarting do not contain all the files

2

if I load after the benchmark is completed from the state file and `bench` again I have `results` being an empty list. As a result, also the upload to the...

make sleep times customizable

1

this is perhaps not necessary in this PR, but I think we should make all our sleep times customizable _Originally posted by @kjappelbaum in https://github.com/lamalab-org/chem-bench/pull/657#discussion_r1944277981_

it is not necessarily obvious if users should expect a new state file or updates to the old one

`dev` is now our new default branch and the actions should not only run on `main`

https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files

consider making the full clone faster

2

if we do the full `gh repo clone lamalab-org/chem-bench` it is still super slow. Fixing this will require rewriting git history, I guess (so this is a drawback). But right...

implement prompt management system

2

- there are tons of people trying to build LLM-ops tooling. https://langbear.runbear.io/application/prompt-templates seems to be one of the most lightweight ones. But most of them come with some GUI or...

write docs explaining the main abstractions in chembench

2

I feel we now have quite a number of abstractions in `ChemBench`, and perhaps some docs explaining them (e.g., the `ChemBenchmark`, `Prompter`, `TopicRegistry`, `BenchmarkState`) would help create some mental scaffolding...