Workaround selfhost Windows

[tuyuribr]

Hi I finally got it working and I'm going to share my step by step to make this work.

My system:

RTX 3060 12GB CUDA 12.1 Windows 10 PHPSTORM 2023.2.4

Step 1 - Install TGI

Follow the step by step here: https://huggingface.co/docs/text-generation-inference/quicktour and make/test your docker

I'm using this rn: docker run --gpus all --shm-size 1g -p 8080:80 -v C:/data:/data ghcr.io/huggingface/text-generation-inference:1.3 --model-id TheBloke/CodeLlama-7B-GPTQ --quantize gptq Test it using: curl http://127.0.0.1:8080/generate -X POST -d "{\"inputs\":\"What is Deep Learning?\",\"parameters\":{\"max_new_tokens\":20}}" -H "Content-Type: application/json"

Step 2 - Install the plugin

Install and configure, don't forget to update the model to the docker url: Also set the log level to info.

If something is missing/not working you can edit the config later here : %appData%/JetBrains/ MY IDE /options/LlmSettingsPlugin.xml

Step 3 - PATH Hell

Right now the plugin ignore the llm-ls path setting on startup, so it always try to re-install from github, and to do so it is using some commands that not all windows machines have https://github.com/huggingface/llm-intellij/blob/2972e337497cd3df9325f0c10a231693ef8ea13d/src/main/kotlin/co/huggingface/llmintellij/lsp/LlmLsLspServerDescriptor.kt#L84-L88 So configure a new folder into your system path (mine C:/mybins/ ) Download/grab a random .exe ( please, don't use doom ), I grabbed the gzip windows bin here: https://gnuwin32.sourceforge.net/packages/gzip.htm . And duplicate for these commands: chmod,gunzip,mv,rm Like this:

Step 4 - Restart the IDE and logs

Restart your IDE and open the logs: %LOCALAPPDATA%/JetBrains/ MY IDE /log/idea.log Look for something like this:

2024-01-07 01:46:07,650 [   4200]   WARN - #c.i.p.l.i.LspServerImpl - LlmLsLspServerDescriptor@aaa(RUNNING;0): Failed to start server
com.intellij.execution.process.ProcessNotCreatedException: Cannot run program "C:\Users\p\.cache\llm_intellij\bin\llm-ls-amd64-pc-windows-msvc-0.4.0": CreateProcess error=2, O sistema não pode encontrar o arquivo especificado
	at com.intellij.execution.configurations.GeneralCommandLine.createProcess(GeneralCommandLine.java:356)
	at com.intellij.execution.process.OSProcessHandler.startProcess(OSProcessHandler.java:84)
	at com.intellij.execution.process.OSProcessHandler.<init>(OSProcessHandler.java:44)
	at com.intellij.platform.lsp.api.LspServerDescriptor.startServerProcess(LspServerDescriptor.kt:95)
	at com.intellij.platform.lsp.impl.connector.Lsp4jServerConnectorStdio.<init>(Lsp4jServerConnectorStdio.java:19)

Get the file name (in this case llm-ls-amd64-pc-windows-msvc-0.4.0 ), rename the current llm-ls bin (the .exe file in the .zip) to match this file ( v0.4.0 Here ). Close the IDE, and put the new file in %HOME%/.cache/llm_intellij/bin/ (or whatever path the log indicates )

Step 5 - Open the IDE and test

Open a file, type something and wait ~3 seconds. It should trigger a request to the docker api. You can check too in the llm-ls log %HOME%/.cache/llm_ls/llm-ls.log. It should look like this in the log:

And like this in the IDE:

Important notes

Right now there isn't so much info about the plugin, and this guide is a workaround for testing the plugin. I didn't find a hotkey to ask to generate, or to make an manual input if you find it please tell me.

Good luck

[Changesong]

I ran the method as written (but API server is external) and now I don't get any errors, but I only get the message below and it doesn't generate any code.

2024-01-17 19:19:16,101 [  40408]   INFO - #c.i.p.l.i.LspServerImpl - LlmLsLspServerDescriptor@test_src(RUNNING;10): {llm-ls} file changed

The request never arrived on the server. llm-vscode works fine, so it doesn't seem to be a server issue. Something doesn't seem to be sending the modifications to the server. Does anyone else have the same problem?

[tuyuribr]

I ran the method as written (but API server is external) and now I don't get any errors, but I only get the message below and it doesn't generate any code.

2024-01-17 19:19:16,101 [  40408]   INFO - #c.i.p.l.i.LspServerImpl - LlmLsLspServerDescriptor@test_src(RUNNING;10): {llm-ls} file changed

The request never arrived on the server. llm-vscode works fine, so it doesn't seem to be a server issue. Something doesn't seem to be sending the modifications to the server. Does anyone else have the same problem?

Can you print the config?

What appear in the llm-ls.log ?

[Changesong]

LlmSettingsPlugin.xml

<application>
  <component name="co.huggingface.llmintellij.LlmSettingsState">
    <option name="fim">
      <FimParams>
        <option name="middle" value=" &lt;MID&gt;" />
        <option name="prefix" value="&lt;PRE&gt; " />
        <option name="suffix" value=" &lt;SUF&gt;" />
      </FimParams>
    </option>
    <option name="lsp">
      <LspSettings>
        <option name="binaryPath" value="C:\Users\changesong\.cache\llm_intellij\bin\llm-ls-amd64-pc-windows-msvc-0.4.0" />
        <option name="logLevel" value="info" />
      </LspSettings>
    </option>
    <option name="model" value="http://localhost:8000/api/generate/" />
    <option name="tlsSkipVerifyInsecure" value="true" />
    <option name="tokensToClear">
      <list>
        <option value="&lt;EOT&gt;" />
      </list>
    </option>
  </component>
</application>

Use port forwarding to forward requests to localhost to the server. Currently, the server is not receiving any requests.

llm-ls.log

{"timestamp":"2024-01-18T02:50:32.481981Z","level":"INFO","message":"received completion request for file:///C:/Users/changesong/Downloads/test/test.cpp","document_url":"file:///C:/Users/changesong/Downloads/test/test.cpp","cursor_line":"8","cursor_character":"13","language_id":"cpp","model":"http://localhost:8000/api/generate/","ide":"jetbrains","max_new_tokens":60,"temperature":0.20000000298023224,"do_sample":true,"top_p":0.949999988079071,"stop_tokens":"None","target":"llm_ls","line_number":556,"spans":[{"request_id":"7e077c74-3279-4e8a-8da9-a8e6a46e297d","name":"completion_request"}]}
{"timestamp":"2024-01-18T02:50:32.482012Z","level":"INFO","message":"completion type: SingleLine","completion_type":"single_line","target":"llm_ls","line_number":582,"spans":[{"request_id":"7e077c74-3279-4e8a-8da9-a8e6a46e297d","name":"completion_request"}]}
{"timestamp":"2024-01-18T02:50:32.482384Z","level":"ERROR","err_msg":"파일 이름, 디렉터리 이름 또는 볼륨 레이블 구문이 잘못되었습니다. (os error 123)","target":"llm_ls","line_number":248,"spans":[{"request_id":"7e077c74-3279-4e8a-8da9-a8e6a46e297d","name":"completion_request"}]}

'파일 이름, 디렉터리 이름 또는 볼륨 레이블 구문이 잘못되었습니다.' is 'The filename, directory name, or volume label syntax is incorrect' in Korean

Thanks to you, I found the error message. When using the Korean environment, many programs will throw that error if there is Korean in the path. But this time, I got that error even though I don't have a Korean path, so I need to figure out the problem.