Issue: `evaluate` is very slow (because `get_git_info`)

[davidgilbertson]

Issue you'd like to raise.

I've noticed that evaluate is taking over a minute and a half to run on a single example. I eventually tracked it down to the fact that LangSmith is calling some git commands in the background and this takes about 1m:28s (the actual evals take about 2 seconds).

Suggestion:

Is this really necessary? I should be able to run evals without involving git, no? Even if I'm an outlier with my very slow git repo, it still seems like overhead that could be avoided.

I don't want all that git info as my metadata anyway, I would like a way to opt out (or in).

If you're wondering why my git is so slow, it's because I'm using WSL2 + with files on Windows which is known to be slow with git. It's perfectly fine most of the time, but things like git status take a while. This is the first time I've come across a Python package that called git, so had never really noticed the problem until now.

[hinthornw]

Fun! Thanks for flagging! Agreed with the feedback - will make some changes

[willemmulder]

Has this been fixed? I still see that an evaluator takes a second to run, even when it's only returning just returning a hard value, which should be run in under a millisecond.

[hinthornw]

For which SDK version?

[willemmulder]

"langsmith": "^0.2.13",

Is what I currently have. Not sure if that is the latest.

[hinthornw]

Evaluator timing would be different than the git call - what concurrency are you returning? And what are you using to clock the time? Any example code?

[willemmulder]

Hey @hinthornw sorry we're not using evaluations at the moment, so can't verify. Let's assume it was unrelated to the post from OP👍