Kevin M Jablonka
Kevin M Jablonka
examples: - how would you separate a mixture of ethanol and water - how would you distinguish xy from xz?
e.g. https://www.notion.so/error-8e857f1f0caa48f09edf68eb0ed62ee2?pvs=4#09463ace9a2f4480bdc8d7c629ea5a13 this is due to the LLM step I guess. We could avoid this if we, when we do the aggregation, have a check that the output contains something.
this might allow us to get some errorbars for MCQ it is related to #227
we should allow people to add their model to it, similar to - https://matbench-discovery.materialsproject.org/ - https://matbench.materialsproject.org/Leaderboards%20Per-Task/matbench_v0.1_matbench_perovskites/ - https://mofdscribe.readthedocs.io/
the regex pipeline might now do the following `[ANSWER] E, ICP-OES [/ANSWER]` was parsed to E C E `"[ANSWER] E, DESI-mass spectrometry [/ANSWER]",` was parsed to "E", "D","E", "a", "e",...
## Ressources - UK past papers https://edu.rsc.org/download?ac=515421 - German past paper https://www.scienceolympiaden.de/icho/material/aufgabenarchiv-icho - 2022 IChO China https://www.icho2022.cn/23262/list.htm ## Approach - Focus questions that can be easily converted into multiple choice...
``` No pattern matched in the model output: I apologize, I should not provide any recommendations or information to assist with isolating explosive materials. 2024-01-01 17:22:41.594 +01:00 UTC | WARNING...
- most of the prompt will be fixed in this case - regex must be customized
I now compared mistral and llama ``` The following is a question about chemistry. Answer by responding with the letter(s) of the correct answer(s). Separate multiple answers with a comma....
`[ANSWER]10,000[/ANSWER]` is parsed as `10` and not `10 000`