add partial eval platform
Structure of this is as follows:
- The snippetization notebook is for loading the longmemeval data set and doing a one time reformatting of the LME data set into "snippets". This notebook saves all of this data, which is very large, into a newly created folder, which is git-ignored
- The mini-dataset creation notebook is for filtering and formatting some specific "mini-dataset" which might be for instance 10 snippets. This notebook is also responsible for calling the utility function for ingesting and labeling, since part of the mini data set itself includes GPT-4o labels. The mini-dataset will have two forms: the internal eval CSV, used for the actual CLI test, and then the CSV used for human labeling, which will ultimately be imported into a Google sheet
- The eval_extract_nodes file is for the CLI test. It is currently outdated and will eventually be updated to use the same ingestion and labeling utility function.
[!IMPORTANT] Add partial evaluation platform for dataset processing and graph building evaluation with new classes, functions, and tests.
- Behavior:
- Adds
EvalAddEpisodeResultsclass ineval.pyto compare baseline and candidate graph extractions.- Implements
eval_add_episode_results()function ineval.pyto evaluate graph data quality.- Introduces
create_episodes_from_messages()inutils.pyto generate episodes from messages.- Implements
ingest_and_label_snippet()andingest_and_label_minidataset()inutils.pyfor processing datasets.- Data Handling:
- Adds
lme_dataset_filtered.csvfor storing filtered dataset examples.- Implements data preparation and processing in
eval_extract_nodes.pyusing CSV data.- Testing:
- Adds
eval_e2e_graph_building.pyfor end-to-end graph building evaluation.- Sets up integration tests in
eval_extract_nodes.pyandpytest.ini.- Introduces logging setup in
utils.pyfor test output.This description was created by
for b35729643d9eb694e457f58f895acc4a70e28d11. It will automatically update as commits are pushed.
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.
I have read the CLA Document and I hereby sign the CLA
You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.