vscode-copilot-release
vscode-copilot-release copied to clipboard
microsoft
Inaccurate Line Numbers Returned for Line of Code
- VS Code Version: 1.80.0-insider
- OS Version: Windows 10
Steps to Reproduce:
- Ask the chat what line of code performs a particular function. (Chat will likely get this correct)
- Follow-up by asking what line number that particular line of code is on. (The answer will likely be significantly off.)
It's simple to reproduce:
My knowledge on this codebase is limited, but two potential reasons come to my mind:
- It's from post processing steps before the AI get's to see the code. That should be reasonable to fix.
- The AI model being used generally is incapable of / not trained on line numbers and would require fine-tuning and not a fix in this repo.
Edit: I just noticed that it didn't reference the right class when getting the line number, it was still the wrong one.
One thing that's causing confusion to the model is the usage of "soft line breaks" (Ux21B5), e.g. holding shift while pressing enter.
- VS Code Version: 1.80.0-insider
- OS Version: Windows 10
Steps to Reproduce:
- Ask the chat what line of code performs a particular function. (Chat will likely get this correct)
- Follow-up by asking what line number that particular line of code is on. (The answer will likely be significantly off.)
I’m having the same issue. If I ask copilot what is wring with a certain line it references a different one but worse if i ask it about highlighted lines or even in the inline chat it seems to be looking at other lines and gives nonsensical suggestions. Is there any known fix to this?
To test, I tried using Continue extension for VS Code for mac instead of copilot and had the same problem. I also tried using PyCharm and got the same issue, and writing new scripts from scratch. There seems to be something wrong with how the IDE is parsing the reference code when sending to the generative model.
If this helps anyone, I found a solution that works for me. I wasn't able to get Github copilot working and resolve the above issues but I have gotten the Continue copilot extension working well by using (command+L) to add selected code to the copilot chat. Continue seems basically identical to Github Copilot in functionality except you can choose which LLM model you want to use – from ChatGPT4 to a range of open source ones. https://continue.dev/docs/quickstart
Curious about what cases you feel are hindered by incorrect line numbers being reported by the chat?
Hey, well any use of copilot basically. For me it would respond as though being given any random lines of code from the script rather than the selected ones and autocomplete was just a jumble of nonsense. I couldn’t resolve the issue so switched to Continue and it seems to work ok. It seems to be an uncommon but persistent problem for some users and a bug.
Hmmm, it should provide the answers on the appropriate selection but may get the numbers wrong as the snippet is injected into the prompt without line numbers. Do you have a non line number oriented use case?
I don't think I'm explaining myself well. The issue renders copilot completely useless. If I select a code block, copilot reads the context around it completely incorrectly so it gives completely incorrect suggestions/answers. Also the inline suggestions are just rubbish. It is as though copilot is mixing up all the lines of code in what it is sending to the generative model. I tried everything short of reinstalling my operating system to get it working, gave up and switched to Continue, which is basically identical to copilot but works on my system and lets you choose which generative model to use. It is a bit glitchy at times too (still in development) but basically works.
@JamesAllerlei Do you have any concrete examples that I can reproduce and debug on to improve the product? Or have you just given up?
Hi, yes, sure. Here is an example. Another issue is that it would say I had not provided any lines of code when you can see in the screen shot, I had used @workspace and selected code. I can provide whatever you need. What is most useful?
@JamesAllerlei Is this a large file? This is likely https://github.com/microsoft/vscode-copilot-release/issues/1101
What happens is that we run out of tokens and end up dropping the selection. We do need to work on smarter shrinking of the selection in these cases, but having a more concise selection may help you in this case