eval-dev-quality
eval-dev-quality copied to clipboard
Infer if a model actually returned source code
https://github.com/symflower/eval-dev-quality/pull/39#discussion_r1568814274
If the model does not respond with a code tag, we cannot be sure if the response contains source code until the execution runs. Ideally we would like to infer if we got source code right within GenerateTestsForFile.