temp buffer with Unicode problem

[kotatsuyaki]

Recently I found out that when using emacs-ycmd, if a javascript file contains CJK characters, then each time when I open the file, emacs keeps asking me to Select coding system (default raw-test): . The interesting part is that it's not the javascript file itself (otherwise I know how to configure it) but a *temp file* buffer, which leads me to believe that this is triggered by ycmd.

The screenshot below gives an idea of what's happening:

The *temp file*'s content is like this:

{"event_name":"FileReadyToParse","file_data":{"/home/user/Documents/test-emacs-ycmd-unicode/test.js":{"contents":"console.log(\"你好\");\n","filetypes":["javascript"]}},"filepath":"/home/user/Documents/test-emacs-ycmd-unicode/test.js","line_num":1,"column_num":22}

This is part of the *ycmd-server* output, which I think is related to the issue.

2019-06-28 19:22:52,649 - DEBUG - No global extra conf, not calling method YcmCorePreload
serving on http://localhost:38119
2019-06-28 19:22:52,678 - INFO - Received event notification
2019-06-28 19:22:52,679 - DEBUG - Event name: BufferVisit
2019-06-28 19:22:52,680 - INFO - Using node binary from: /usr/bin/node
2019-06-28 19:22:52,680 - INFO - Not using Tern completer: not installed at /home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/third_party/tern_runtime/node_modules/tern/bin/tern
2019-06-28 19:22:52,680 - INFO - Using TypeScript completer with /home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/third_party/tsserver/bin/tsserver
2019-06-28 19:22:52,681 - INFO - TSServer log file: /tmp/tsserver_8o2hyplj.log
2019-06-28 19:22:52,683 - INFO - Enabling TypeScript completion
2019-06-28 19:22:52,698 - INFO - Received event notification
2019-06-28 19:22:52,699 - DEBUG - Event name: FileReadyToParse
2019-06-28 19:22:52,699 - INFO - Adding buffer identifiers for file: /home/user/Documents/test-emacs-ycmd-unicode/test.js
2019-06-28 19:22:54,682 - INFO - Received typingsInstallerPid event from TSServer
2019-06-28 19:22:55,769 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 261, in _ReadMessage
    key, value = utils.ToUnicode( headerline ).split( ':', 1 )
ValueError: not enough values to unpack (expected 2, got 1)
2019-06-28 19:22:55,770 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 261, in _ReadMessage
    key, value = utils.ToUnicode( headerline ).split( ':', 1 )
ValueError: not enough values to unpack (expected 2, got 1)
2019-06-28 19:22:55,771 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 261, in _ReadMessage
    key, value = utils.ToUnicode( headerline ).split( ':', 1 )
ValueError: not enough values to unpack (expected 2, got 1)
2019-06-28 19:22:55,771 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 261, in _ReadMessage
    key, value = utils.ToUnicode( headerline ).split( ':', 1 )
ValueError: not enough values to unpack (expected 2, got 1)
2019-06-28 19:22:55,772 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 261, in _ReadMessage
    key, value = utils.ToUnicode( headerline ).split( ':', 1 )
ValueError: not enough values to unpack (expected 2, got 1)
2019-06-28 19:22:55,772 - ERROR - Error while reading message from server
Traceback (most recent call last):
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 227, in _ReaderLoop
    message = self._ReadMessage()
  File "/home/user/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/completers/typescript/typescript_completer.py", line 268, in _ReadMessage
    raise RuntimeError( "Missing 'Content-Length' header" )
RuntimeError: Missing 'Content-Length' header
2019-06-28 19:22:55,818 - INFO - Received setTypings event from TSServer
2019-06-28 19:22:56,094 - INFO - Received projectsUpdatedInBackground event from TSServer

• OS: Manjaro Linux 4.19.45-1-MANJARO
• Emacs version: (stock emacs) GNU Emacs 26.2
• emacs-ycmd: 20190416.807, installed via MELPA

[Fyrbll]

I can reproduce this error on the following system:

• OS: Ubuntu 18.04.2
• Emacs version: 25.2.1
• emacs-ycmd: 20190416.807, installed via MELPA

After looking at the source code for a bit, I think I get what's happening.

If you look at the function ycmd--get-basic-request-data, defined in ycmd.el, you'll notice that it's constructing a server request that has the field "file_data" which in turn has the field "contents". The data accompanying this field name is defined using ycmd--encode-string. Said function uses (encode-coding-string s 'utf-8 t).

Problems arise when you try to insert the return value of (encode-coding-string s 'utf-8 t) into a temp buffer when s is string containing CJK characters. As an example, run the following code in your *scratch* buffer:

(with-temp-file (make-temp-file "dummy-file")
  (insert (encode-coding-string "你好" 'utf-8 t)))

It raises a very similar warning.

Here's a guess. The characters 你 and 好 are multi-byte, and when you run toggle-enable-multibyte-characters in your buffer they show up as "\344\275\240" and "\345\245\275" respectively, which matches the numbers in your screenshot. The temporary file probably doesn't support multi-byte characters and then shoots a warning. I feel justified in saying this because running the following code doesn't give me any warning:

(with-temp-file (make-temp-file "dummy-file")
   (toggle-enable-multibyte-characters)
   (insert (encode-coding-string "你好" 'utf-8))
   )

[telotortium]

This is very similar to #387 from a few years ago. A workaround that avoids the issue entirely is changing the request backend away from curl: (setq request-backend 'url-retrieve). However, url-retrieve has its own bugs, so I hope there's a fix soon.

[telotortium]

Here's a better workaround - use advice to change the request backend just for YCMD:

(defun my-ycmd--request-url-retrieve (f &rest args)
  "Change ‘request-backend’ in ‘ycmd--request’ to ‘url-retrieve’.

Work around https://github.com/abingham/emacs-ycmd/issues/496.
"
  (let ((request-backend 'url-retrieve))
    (apply f args)))
(advice-add 'ycmd--request :around #'my-ycmd--request-url-retrieve)

[telotortium]

This should be fixed by https://github.com/tkf/emacs-request/pull/136, and the workaround is no longer needed after that PR. Please close the issue if you upgrade request and no longer see this issue.