Add deepseek-coder api

[FredericoPerimLopes]

Hello, I was able to make the deepseek-coder api work with Aider. Although it works, it might not follow all the prompts. If interested I can make a PR.

[paul-gauthier]

Thanks for trying aider and filing this issue.

It looks like a user benchmarked deepseek-coder-7b-instruct-v1.5_8bit in January. It only scored 44% on the benchmark, which is significantly worse than gpt-3.5-turbo.

https://discord.com/channels/1131200896827654144/1133060780649087048/1201462830835175524

There's also a FAQ entry that might contain helpful information: https://aider.chat/docs/faq.html#can-i-use-aider-with-other-llms-local-llms-etc

[FredericoPerimLopes]

@paul-gauthier . The one I'm talking about is from deepseek.com, I guess it's the 33B models they're using. Their apis are compatible with openai.

[amacsmith]

I would look to

https://huggingface.co/bartowski/stable-code-instruct-3b-GGUF or https://github.com/OpenCodeInterpreter/OpenCodeInterpreter

[paul-gauthier]

You should be able to connect to the deepseek.com api with aider if it is openai compatible. Here are some relevant docs:

https://aider.chat/docs/llms.html#openai-compatible-apis

I just tried the deepseek-coder model like this, and it did ok with the "whole" edit format. It can't handle the diff or udiff edit formats.

export OPENAI_API_KEY=...
export OPENAI_API_BASE=https://api.deepseek.com/v1
aider --model openai/deepseek-coder

[paul-gauthier]

I just benchmarked deepseek-coder on the deepseek.com api. They offer 5M free tokens of usage. It's similar to GPT-3.5 and can only handle the "whole" edit format.

test-cases: 132
model: openai/deepseek-coder
edit_format: whole
commit_hash: c07f793-dirty

error_outputs: 0
user_asks: 0
lazy_comments: 2
num_malformed_responses: 0
syntax_errors: 13
indentation_errors: 0

exhausted_context_windows: 0
test_timeouts: 2

47.0% correct after try 0
54.5% correct after try 1

duration: 26.7 sec/test-case

[younes-io]

I just benchmarked deepseek-coder on the deepseek.com api. They offer 5M free tokens of usage. It's similar to GPT-3.5 and can only handle the "whole" edit format.

test-cases: 132
model: openai/deepseek-coder
edit_format: whole
commit_hash: c07f793-dirty

error_outputs: 0
user_asks: 0
lazy_comments: 2
num_malformed_responses: 0
syntax_errors: 13
indentation_errors: 0

exhausted_context_windows: 0
test_timeouts: 2

47.0% correct after try 0
54.5% correct after try 1

duration: 26.7 sec/test-case

How poor is its performance compared to gpt4-turbo-preview?

EDIT: I'm asking because the pricing is just crazily cheap! It beats everyone out there.. in other words, I'm willing to re-try 10 times with DeepSeek because it's just super cheap (ofc, as long as it's able to adjust itself) cf. https://github.com/deepseek-ai/DeepSeek-V2?tab=readme-ov-file#6-api-platform

[paul-gauthier]

See the LLM leaderboards for more comparisons. Opus is the best. GPT-4 Turbo is close behind it. Deepseek coder is a reasonable alternative, and is extremely inexpensive. But it's not as good as those two and has a more limited context window.

https://aider.chat/docs/leaderboards/

[younes-io]

See the LLM leaderboards for more comparisons. Opus is the best. GPT-4 Turbo is close behind it. Deepseek coder is a reasonable alternative, and is extremely inexpensive. But it's not as good as those two and has a more limited context window.

https://aider.chat/docs/leaderboards/

Thank you, Paul