Question about possible missing file in the demonstration prompts when running the benchmark

[nims11]

Issue

I am running the aider benchmark and noticed the first user turn is possibly referring to source which the model hasn't seen?

Here is the dump of the message sent to the LLM (Change the greeting to be more casual is the first user message). https://gist.github.com/nims11/6e172deb100154b54bb96a259cf66acb

Looking at the prompts in some other diff formats they also follow a similar pattern. Is this intended behaviour?

Version and model info

No response

[paul-gauthier]

Thanks for trying aider and filing this issue.

Sorry, what exactly is the problem you are reporting?

[nims11]

My understanding is that for the aider benchmark, there are few shot examples (as in aider/coders/editblock_fenced_prompts.py) before letting the model generate the answer.

My question is around these few shot examples / demonstrations. In editblock_fenced_prompts.py, the first user message is Change get_factorial() to use math.factorial, which is a bit odd considering there is no initial source to modify. I am not sure how much this affects actual model performance, but I wanted to confirm if this was an intentional choice of prompts.

[paul-gauthier]

Ah yes, fair point. The models seem to do well with these prompts. But it would be interesting to expand the example to include the "pre existing" source code, as that is how it would work in a real aider chat.

[github-actions[bot]]

This issue has been labelled stale because it has been open for 2 weeks with no activity. Remove stale label or add a comment to keep this issue open. Otherwise, it will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 3 weeks with no activity. Feel free to add a comment here and we can re-open it. Or feel free to file a new issue any time.