stagehand icon indicating copy to clipboard operation
stagehand copied to clipboard

use string similarity on extract_regulations eval

Open seanmcguire12 opened this issue 11 months ago • 1 comments

why

  • target website has a typo, LLM corrects the typo when printing extracted data

what changed

  • use string similarity to allow for small differences. we pass with/without the typo

seanmcguire12 avatar Dec 31 '24 00:12 seanmcguire12

⚠️ No Changeset found

Latest commit: 73cc5ec92a2ca06d61696a96c6255badf6d192b6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

changeset-bot[bot] avatar Dec 31 '24 00:12 changeset-bot[bot]

don't think this is necessary; would rather make extract promptable to avoid spelling errors

kamath avatar Dec 31 '24 03:12 kamath