bug: "Generate Tests" generates a valid test but missing `import` statement

[noah-berman]

Version

v1.15.1714144471 (pre-release)

Describe the bug

Customer has reported that, for both sample TypeScript and Python files, "Generate Tests" has correctly populated valid unit tests, but instead of adding an import statement at the top of the file it simply lists the path of the new test file:

Expected behavior

Test file should have an import statement to correctly import the function/method in question

Additional context

Customer is using Cody Pro with freshly spun-up TypeScript and Python repositories for testing. Client has tried switching LLMs across Sonnet, Opus, GPT-3.5 Turbo, and GPT-4 Turbo

[sqs]

Notes:

• Should we run the VS Code "add all missing imports" action after generating a test? Might get it wrong but it's usually good.
• The inclusion of a file path at the top is a bug.

[umpox]

• Support running "add missing imports action"
• Separate file path prompt into a separate request. (Right now we try to get the LLM to provide both the file name and the contents in the response, this is error prone)
• The fact it isn't adding any imports at all makes me think this is potentially a prompting issue. It could be that we're not telling the LLM that we're in a new file, so it's assuming we already have access? I think we can fix this reliably

[abeatrix]

@umpox we have a newTestSuitePrompt that specifically asked the LLM not to include any imports if there is an existing test file, since the new test suite is appended to the file. Maybe we can remove it to support "add all missing imports"?

[github-actions[bot]]

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.