zed
zed copied to clipboard
zed-industries
copilot chat doesn't use chatGPT 4
Check for existing issues
- [X] Completed
Describe the bug / provide steps to reproduce it
the two modes for the new copilot chat ( GPT-4 and GPT-3.5) don't seem to make any difference, I asked both of them which model of gpt they are using and they both replied with GPT-3, even when picking the GPT-4. I tried the same thing with the same account in vscode and it returned that it uses GPT-4o. ( as seen in the pictures below ) is this a bug? or something that I did wrong? anyway I have been using zed ever since its release on linux and it has been amazing. great work.
Environment
Zed: v0.147.2 (Zed) OS: Linux Wayland arch unknown Memory: 15.3 GiB Architecture: x86_64 GPU: Intel(R) Graphics (ADL GT2) || Intel open-source Mesa driver || Mesa 24.1.5-arch1.1
If applicable, add mockups / screenshots to help explain present your vision of the feature
If applicable, attach your Zed.log file to this issue.
Zed.log
I can reproduce this with my GitHub account. I also get the following response:
I am based on OpenAI's GPT-3 model.
I'm not sure just how much I'd trust the model here, as it very well could be hallucinating or just plain wrong. If you look through the code, you can see that the GitHub API is indeed being called with different models, and there's not anything else that Zed can do beyond that. A better test would be to see how the different models perform on the same task, as GPT-4 should be significantly better. Regardless, I've just created a merge request to enable GPT-4o functionality (#16446 ) which I can confirm from my testing is definitely being used. The 128K context window and much faster response time is absolutely wonderful for asking questions in an unfamiliar codebase and confirms the API is working as intended.
Never trust the output of the model if you ask it about itself. If the system prompt is not updated correctly, you will get wrong information.
To test if you are using GPT-3 or GPT-4 use the following prompt, which is with ~9 out of 10 only correctly solved by GPT-4: Give me 10 sentences that end with the word "apple"
I'm going to go ahead and close this. In the intervening two months I've become accustomed to models hallucinating other models as their own name, even when the correct name is explicitly defined in the system prompt, so this is no longer as surprising to me that it could get this wrong.