Failed to launch Continue with llama_cpp_python server

[vlsav]

Before submitting your bug report

• [ ] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [ ] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 10
- Continue: 0.9.43
- IDE: VSCode 1.80.1

Description

Configured two models with config: "models": [ { "title": "DeepSeek-1b", "model": "DeepSeek-1b", "completionOptions": {}, "provider": "openai", "apiBase": "http://localhost:8000/v1" }, { "title": "mistral-7b", "model": "mistral-7b", "completionOptions": {}, "provider": "openai", "apiBase": "http://localhost:8000/v1" } ], ... launching server by python -m llama_cpp.server --model "path to local deepseek or mistral GGUF" --use_mlock True --n_ctx 2048 launched without issues. Choose model in Continue, entering any question, Enter server console shows POST /v1/chat/completions HTTP/1.1 200 OK Exception i callback _ProactorBasePipeTransport._call_connection_lost(None) ... ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host disconnected Disconnected from client (via refresh/close) ...

In Continue it shows Error: The request failed with "ReferenceError": TransformStream is not defined

To reproduce

No response

Log output

in Logs it it the same:
The request failed with "ReferenceError": TransformStream is not defined
notificationsAlerts.ts:42

[sestinj]

@vlsav We don't directly use TransformStream which leads me to believe this is a Node version compatibility issue (it was only introduced in Node version 18 as noted here). Would you mind checking the Node version that you VS Code uses? This should be displayed in the same menu that shows the VS Code version.

Assuming that this is the case, the current workaround would be to upgrade your Node and/or VS Code version.

In the next few days a set of changes I am making to use node-fetch instead of undici should resolve this for good

[vlsav]

Node.js version is 16.17.1, so it is a root of the issue. Unfortunately, there is no chance to update the vscode in the near future.

[jonfairbanks]

I also had this issue and resolved in by updating my copy of VSCode.

Fwiw, I am also running Node v18.16.0.

[sestinj]

@vlsav thanks for sharing. I'd be curious to know why if you're able to share (and I really mean this out of curiosity only, because if it turned out to be super common, then we would start paying more attention to version compatibilities)

I'm getting close to the fix that may or may not solve the issue anyway, hopefully have an update tomorrow

[vlsav]

I don't have admin rights on that PC. So just Python (reason for llama-cpp-python) and existing vscode. Hopefully, it will be upgraded sooner or later, but I am not able to do it myself. I am still able to do it on another PC, but if there is no dependency, it will be helpful if it would be possible to use older vscode as well. Thank you for support!

[sestinj]

@vlsav If you're interested in giving it a try, we just switched to using node-fetch, which seems to be compatible all the way back to Node 12 (https://www.npmjs.com/package/node-fetch#:~:text=current%20stable%20release%20(3.x)%20requires%20at%20least%20node.js%2012.20.0.). There's at least some chance this would solve the problem

[vlsav]

Upgraded to latest version. Strangely my deepseek and mistrial disappeared from model selection. I see notification: Error loading config.json. please check your config.json file: Error: Build Error. Please check your ~/.continue/config.ts file: Error: spawn UNKNOWN

[surak]

Upgraded to latest version. Strangely my deepseek and mistrial disappeared from model selection. I see notification: Error loading config.json. please check your config.json file: Error: Build Error. Please check your ~/.continue/config.ts file: Error: spawn UNKNOWN

Please check #828

[sestinj]

@vlsav there was a problem loading the config in dev containers that is now solved in 0.9.52. does this solve your problem, or might something else be happening?

[vlsav]

Upgraded to latest version 0.9.53. Same story, no changes: "Strangely my deepseek and mistrial disappeared from model selection. I see notification: Error loading configuration.json. please check your config.json file: Error: Build Error. Please check your ~/.continue/config.ts file: Error: spawn UNKNOWN"

I also tried to uninstall, remove old ~/.continue/* then install again. I am encountering the same error even without adding deepseek, mistral local models because I can't add them now due to this error.

[mac5yver]

Upgraded to latest version. Strangely my deepseek and mistrial disappeared from model selection. I see notification: Error loading config.json. please check your config.json file: Error: Build Error. Please check your ~/.continue/config.ts file: Error: spawn UNKNOWN

Same issue here. Can't use Continue because I can't modify and save config.json with error: "Error loading config.json. Please check your config.json file: Error: Build error. Please check your ~/.continue/config.ts file: Error: spawn UNKNOWN"

Windows 11 22H2 Continue Version: 0.9.62 VS Code

[vlsav]

With continue version 0.9.73 no issues. It just took my existing config with deepseek and communicates with local OpenAI endpoint of llama_cpp_python without issues. Results itself a bit confusing, but working. So closing this issue