graphiti icon indicating copy to clipboard operation
graphiti copied to clipboard

add partial eval platform

Open jackaldenryan opened this issue 1 year ago • 1 comments

Structure of this is as follows:

  • The snippetization notebook is for loading the longmemeval data set and doing a one time reformatting of the LME data set into "snippets". This notebook saves all of this data, which is very large, into a newly created folder, which is git-ignored
  • The mini-dataset creation notebook is for filtering and formatting some specific "mini-dataset" which might be for instance 10 snippets. This notebook is also responsible for calling the utility function for ingesting and labeling, since part of the mini data set itself includes GPT-4o labels. The mini-dataset will have two forms: the internal eval CSV, used for the actual CLI test, and then the CSV used for human labeling, which will ultimately be imported into a Google sheet
  • The eval_extract_nodes file is for the CLI test. It is currently outdated and will eventually be updated to use the same ingestion and labeling utility function.

[!IMPORTANT] Add partial evaluation platform for dataset processing and graph building evaluation with new classes, functions, and tests.

  • Behavior:
    • Adds EvalAddEpisodeResults class in eval.py to compare baseline and candidate graph extractions.
    • Implements eval_add_episode_results() function in eval.py to evaluate graph data quality.
    • Introduces create_episodes_from_messages() in utils.py to generate episodes from messages.
    • Implements ingest_and_label_snippet() and ingest_and_label_minidataset() in utils.py for processing datasets.
  • Data Handling:
    • Adds lme_dataset_filtered.csv for storing filtered dataset examples.
    • Implements data preparation and processing in eval_extract_nodes.py using CSV data.
  • Testing:
    • Adds eval_e2e_graph_building.py for end-to-end graph building evaluation.
    • Sets up integration tests in eval_extract_nodes.py and pytest.ini.
    • Introduces logging setup in utils.py for test output.

This description was created by Ellipsis for b35729643d9eb694e457f58f895acc4a70e28d11. It will automatically update as commits are pushed.

jackaldenryan avatar Jan 10 '25 20:01 jackaldenryan


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

github-actions[bot] avatar Jan 10 '25 20:01 github-actions[bot]