langchain-benchmarks icon indicating copy to clipboard operation
langchain-benchmarks copied to clipboard

Is it possible to download the benchmarks locally

Open Kirushikesh opened this issue 9 months ago • 0 comments

So each sample in the benchmark contains input(query) and output(reference, expected_steps, order_matters) which is visible in the langsmith UI, but when i try to download the same task benchmarks locally using this code

download_public_dataset(
    "https://smith.langchain.com/public/452ccafc-18e1-4314-885b-edd735f17b9d/"
)

with open("./452ccafc-18e1-4314-885b-edd735f17b9d.json", "r", encoding="utf-8") as f:
    print(json.dumps(json.load(f)[0], indent=2, sort_keys=True))

Output:

[
  {
    "created_at": "2024-08-29T18:00:33.682146+00:00",
    "dataset_id": "3f29798f-5939-4643-bd99-008ca66b72ed",
    "id": "1620401b-f41f-4dcc-a2a4-a9026504b2c9",
    "inputs": {
      "question": "What are the benefits of using LangChain for building chatbots?"
    },
    "metadata": {
      "dataset_split": [
        "base"
      ],
      "source": "synthetic"
    },
    "modified_at": "2024-08-29T18:00:33.682146+00:00",
    "outputs": {
      "answer": "LangChain offers several benefits for building chatbots, including modular components for handling various tasks, easy integration with multiple language models, built-in support for memory and state management, and a flexible architecture that allows for customization and scalability."
    },
    "runs": [],
    "source_run_id": null
  }
]

It doesn't contain all the output variables like intermediate_steps. Is there any other way to download the complete benchmark dataset for a task locally?

Kirushikesh avatar Mar 10 '25 11:03 Kirushikesh