CUDA out of memory with the latest update

[stratte89]

Describe the bug

after updating to the latest version i get a CUDA out of memory. Ubuntu, i5 11400, 32gb ram, rtx3080 10gb vram.

both llm were working fine without Cuda oom error before with same settings. Any Idea what changed what could cause this error? Does Devin take more memory now?

My config.toml and environment vars (be sure to redact API keys): LLM_API_KEY="na" LLM_BASE_URL="http://0.0.0.0:5000/v1" LLM_MODEL="openai/Mistral" LLM_EMBEDDING_MODEL="lokal" WORKSPACE_DIR="./workspace" MAX_ITERATIONS=20000

My model and agent (you can see these settings in the UI):

• Oobabooga Web UI
• Model: deepseek-ai_deepseek-coder-1.3b-instruct https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct

TheBloke_Mistral-7B-Instruct-v0.2-code-ft-AWQ https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-code-ft-AWQ

this one works although its way bigger in size than the other Unblocked_Finetuned-Mistral-7B-v0.2-code-full https://huggingface.co/Unblocked/Finetuned-Mistral-7B-v0.2-code-full only problem with this one is, that it stops prompting in the frontend only prompting in the make start-backend terminal without writing any actual code

• Agent: Monologue

Logs, error messages, and screenshots: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 644.00 MiB. GPU 0 has a total capacity of 9.54 GiB of which 490.75 MiB is free. Process 1583 has 0 bytes memory in use. Process 4681 has 414.00 MiB memory in use. Including non-PyTorch memory, this process has 7.90 GiB memory in use. Of the allocated memory 7.36 GiB is allocated by PyTorch, and 282.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Output generated in 1.79 seconds (0.00 tokens/s, 0 tokens, context 5222, seed 2044079865)