Adrian Mirza

https://github.com/lamalab-org/chem-bench/blob/d4abb425aad432fea4a2af8b2df7ffde9fcf10da/tox.ini#L19

The files are formatted inconsistently. Some are just one line.

feat: summary file for multiple models

2

Create a summary file after completion of `.bench`.

feat: add human subset, updated classification, plotting function

12

## Summary by Sourcery Add new functionalities for selecting a human subset of questions and plotting topic distributions. Refactor the script to improve data processing and classification, including keyword filtering...

refactor: move icho to metadata

4

- [ ] change all questions with icho to meta - icho - [ ] add icho model to metadata

Proposing this: ```python @field_validator("name") def check_the_name_is_not_a_duplicate(cls, v): current_absolute_path = Path(__name__).resolve().parent # glob with the pattern of the file name files = current_absolute_path.glob("*.json") # read json files for file in files:...

https://github.com/lamalab-org/chem-bench/blob/a2439b98886356075ca16b5f2494349280e5174b/src/chembench/evaluate.py#L479 Here `is_model_local=True`, while the default model is set to GPT-4

_Originally posted by @kjappelbaum in https://github.com/lamalab-org/chem-bench/issues/509#issuecomment-2320325704_