eval-dev-quality icon indicating copy to clipboard operation
eval-dev-quality copied to clipboard

Infer if a model actually returned source code

Open bauersimon opened this issue 1 year ago • 0 comments

https://github.com/symflower/eval-dev-quality/pull/39#discussion_r1568814274

If the model does not respond with a code tag, we cannot be sure if the response contains source code until the execution runs. Ideally we would like to infer if we got source code right within GenerateTestsForFile.

bauersimon avatar Apr 18 '24 12:04 bauersimon