clangd fails with message "trying to get AST for non-added document"

[yuki1318]

When used with the ConvertToUTF8 plugin Clangd fails to update file when saving file(file encoding is cp932)

Does not occur with the ST3 version of the LSP plugin

language server log clangd: E [09: 41: 12.803] Failed to update "file_name".c: Change's rangeLength (165830) doesn't match the computed range length (159259).

Troubleshooting: clangd

Version

• LSP: 1.7.0
• Sublime Text: 4113

Server Test Run

• exit code: 0
• output

I[09:57:27.986] clangd version 12.0.0
I[09:57:27.987] PID: 7812
I[09:57:27.987] Working directory: D:\work\
I[09:57:27.987] argv[0]: clangd
I[09:57:27.987] argv[1]: --background-index
I[09:57:27.987] argv[2]: --header-insertion=never
I[09:57:27.987] argv[3]: -j=4
I[09:57:27.987] argv[4]: --pch-storage=disk
I[09:57:27.990] Starting LSP over stdin/stdout

[rwols]

What does Sublime Text itself think the encoding is? The ST4 variant of LSP uses incremental text sync to save RPC bandwidth. The ST3 version sends the entire buffer over the RPC layer for each mutation.

If this ConvertToUTF8 package attempts to work with encodings not supported by ST, then I'm afraid the incremental text sync is not going to work.

Convert the files to UTF8 before opening them, or any encoding that ST understands.

[rwols]

There's multiple "real" ways to fix encoding issues:

1. ST should support your missing encodings out of the box.
2. ST can expose an EncodingProvider in the python API and plugins can define new encodings.

The way the ConvertToUTF8 package currently works is by way of circumventing most of the ST API from a quick reading. These kinds of plugins never work well with other listener plugins and synergy is therefore lost.

[yuki1318]

1. ST should support your missing encodings out of the box.

I think there is hopeless that ST support missing encondings...

2. ST can expose an EncodingProvider in the python API and plugins can define new encodings.

Sorry. I can't understand it. You mean ST plugins can add missing encoding to "Set Encoding list"?

The way the ConvertToUTF8 package currently works is by way of circumventing most of the ST API from a quick reading. These kinds of plugins never work well with other listener plugins and synergy is therefore lost.

What should the "ConvertToUTF8" plugin be that LSP plugin does fine???

[rwols]

I did some research. At the end of the day ConvertToUTF8 has:

• 2 calls to view.replace
• 1 call to view.set_encoding in on_pre_save
• 1 call to view.insert, which is used for a scratch buffer and so is irrelevant

I would expect view.replace to pose no problem for the incremental sync feature. Perhaps there is something going wrong in view.set_encoding. I would have to investigate further.

[rwols]

Do you have an example of a simple main.cpp file with Japanese characters in cp932 encoding that exhibits the bug?

[yuki1318]

OK.

main.zip