How to debig semantic highlighting not showing up?

[JoeyEremondi]

Thank you for the bug report

• [X] I am using the latest version of lsp-mode related packages.
• [X] I checked FAQ and Troubleshooting sections
• [X] You may also try reproduce the issue using clean environment using the following command

emacs -q -l lsp-start-plain.el

where lsp-start-plain.el can be downloaded here.

Bug description

I'm looking for advice on how to debug semantic highlighting not being shown for a Language Server I am helping to develop. The highlighting seem to work in other editors, so the problem must be how we're interacting with Emacs somehow.

Steps to reproduce

I'm more looking for help debugging, since I'm not expecting the devs to build and install an the idris2-lsp server, but here's the info:

Prereq: idris2-lsp built and on $PATH: https://github.com/idris-community/idris2-lsp

Config: lsp-start-plain.el with the following lines added

(add-to-list 'lsp-language-id-configuration '(".*\\.idr$" . "idris2"))

    (lsp-register-client
     (make-lsp-client
      :new-connection (lsp-stdio-connection "idris2-lsp")
      :major-modes '(fundamental-mode)
      :server-id 'idris2-lsp))

    (setq lsp-semantic-tokens-enable t)

File: Hello.idr

plus : Nat -> Nat -> Nat
plus x y = x + y

Expected behavior

The file is highlighted with semantic highlighting.

Which Language Server did you use?

There is no highlighting, even though info is being sent from the server.

The server's stderr output shows that this is what's being sent:

LOG INFO:Communication.Channel: Received request for method "textDocument/semanticTokens/full"
LOG DEBUG:Notification.SemanticTokens: Fetching semantic highlightning metadata
LOG DEBUG:Notification.SemanticTokens: Fetching semantic aliases metadata
LOG DEBUG:Notification.SemanticTokens: Fetching semantic defaults metadata
LOG DEBUG:Notification.SemanticTokens: Removing overlapping tokens
LOG DEBUG:Notification.SemanticTokens: Splitting multiline tokens
LOG DEBUG:Notification.SemanticTokens: Encoding semantic tokens
LOG INFO:Communication.Channel: Sent response message for method "textDocument/semanticTokens/full"
LOG DEBUG:Communication.Channel: Response sent: {"jsonrpc":"2.0","id":2,"result":{"data":[0,0,4,4,0,0,5,3,0,0,0,4,1,4,0,0,2,4,0,0,2,0,4,1,0,0,5,1,4,0,0,2,3,0,0,0,4,2,4,0,0,3,3,0,0,0,4,2,4,0,0,3,3,0,0,0,4,2,4,0,0,3,3,0,0,1,0,4,1,0,0,5,1,3,0,0,2,1,3,0,0,2,1,3,0,0,2,1,4,0,0,2,1,3,0,0,2,1,1,0,0,2,1,3,0]}}
LOG DEBUG:Communication.Channel: Received message: {"jsonrpc":"2.0","method":"$/cancelRequest","params":{"id":2}}

OS

Linux

Error callstack

No response

Anything else?

Basically, I'm looking for advice for how to debug this. I was directed here by @ericdallo on Discord, and told to ping @sebastiansturm.

Are there flags I can turn on to see what's happening inside the semantic highlighting, and to see why it's receiving the data but not highlighting based on it? Is there anything obviously wrong in the data being sent? I tried to do a small file but unfortunately it's still a big array.

[sebastiansturm]

I recommend enabling workspace logs using (setq lsp-log-io t) and posting the results here; I'd also trace lsp-semantic-tokens--fontify (via trace-function) to see if it's being invoked at all.

Also, there was a bug a while ago where tokens would not be rendered if they were near to the buffer beginning; while to the extent of my knowledge that bug has been fixed, such an issue might affect test buffers as short as the one you seem to be using. So it might be interesting to use a larger idris2 source file and check if token highlights are missing entirely, or just within certain buffer regions.

As to the idris2 language server -- are there binary builds for x86_64/Ubuntu 20.04 somewhere?

[JoeyEremondi]

I can confirm that lsp-semantic-tokens--fontify is not being called. Tried tracing it for a random Python file, and it was called just fine. So something is preventing it from being called.

I'll look into the compiled binary.

[JoeyEremondi]

I've put a binary and a test source code file here:https://github.com/JoeyEremondi/JoeyEremondi.github.io/tree/main/static/files

That said, I'll totally understand if you don't want to run a random binary from my GitHub.

[yyoncho]

@JoeyEremondi JFYI you can use https://github.com/emacs-lsp/lsp-gitpod

[sebastiansturm]

@JoeyEremondi that's ok; I wouldn't sit down and try to audit the idris2-lsp code before compiling, so I can just as well trust you with the binary I guess. It seems to be incomplete though, or maybe I'm missing something. As far as I can see, the entire zip is around 1.3 KB, and the executable tries to invoke some idris2-lsp_app stored elsewhere?

[onepunchtech]

I'm implementing a language server, and I'm also having trouble getting semantic highlighting. I'm posting here because the name of the issue is how to debug. Let me know if I should create a different issue, or if there is a more appropriate place.

I was testing, trying to highlight a single token. I was changing the location and token type to see if I could get anything to highlight, but nothing is working. [Debug] <--2-- {"id":262,"jsonrpc":"2.0","result":{"data":[1,3,6,15,0]}}

I did trace-function on lsp-semantic-tokens--fontify and this is the output on the start of lsp (even if I return null as token data)

======================================================================
1 -> (lsp-semantic-tokens--fontify #[128 "\301\302\300!\"\207" [font-lock-fontify-region-function apply default-value] 4 "\n\n(fn &rest ARGS)"] 1 233 nil)
1 <- lsp-semantic-tokens--fontify: !non-local\ exit!

[sebastiansturm]

Hi @onepunchtech, I'd try using edebug to set a breakpoint within lsp--semantic-tokens-fontify and then call lsp--semantic-tokens-fontify interactively. That way you should be able to step through the token buffer as it is being processed by lsp-mode. Does that work for you?

[onepunchtech]

@sebastiansturm I've determined that highlighting works with a large enough file. I have to work on some other things first, but I will come back to this and report/fix any bugs I find.

Thanks for the help. I'm new to debugging emacs lisp.

[onepunchtech]

Ok, need a little help. I'm unable to get edebug to trigger when lsp-semantic-tokens--fontify is triggered from my language's buffer. I can enter into debugger if I trigger it manually from another elisp buffer.

These are the steps I followed.

1. clone repo locally
2. add :load-path setting to use-package
3. open lsp-semantic-tokens.el go to lsp-semantic-tokens--fontify definition and call edebug-defun
4. open my language mode file and execute eval-buffer
5. try reloading various edits in file to trigger lsp-semantic-tokens--fontify

I tried various permutations of the above steps and can't get the lsp-mode to trigger edebug. I also verified through trace-function that lsp-semantic-tokens--fontify is getting called.

I can however open scratch buffer for example, and call lsp-semantic-tokens--fontify and it will trigger edebug and I can step through the function. This isn't useful because it doesn't have the same result as my language buffer.

Is there a better way to get edebug to work with lsp-mode?

[sebastiansturm]

not sure if functions called during fontification won't invoke edebug but I suspect this is the case, as errors thrown during fontification don't throw a backtrace either (at least in my experience; I have seen this but haven't looked into what mechanisms are at play here). That being said, in your situation I wouldn't want to wait for automatic fontification anyway but rather call the function manually so I have full control over the supplied parameters. You might also want to define a dummy "old-fontify" function that does nothing but return the fontification bounds in unmodified form:

(defun fontify-do-nothing (beg end loudly)
  ;; just return unmodified bounds
  `(jit-lock-bounds ,beg . ,end))

With that function in place, I'd switch to the buffer you want to investigate, and (after having set a breakpoint within lsp-semantic-tokens--fontify using edebug-set-breakpoint), call fontification manually, probably on the entire buffer:

(lsp-semantic-tokens--fontify #'fontify-do-nothing (point-min) (point-max) t)

That should invoke edebug within lsp-semantic-tokens--fontify and allow you to step through it; at least it worked for me when I just tested it. Hope this helps?

[gitonthescene]

FWIW, I implemented trivial support for semantic tokens for a server implemented with pygls for with a trivial text-mode for an esolang. I also was not getting it to show up. After poking around it looks like lsp-semantic-tokens.el calls font-lock-flush rather than font-lock-ensure. It seems the difference between the two is the check for font-lock-fontified. When I set font-lock-fontified my semantic highlighting suddenly worked.

Should this be set or at least checked for somewhere in lsp-semantic-tokens.el?

[ericdallo]

Makes sense to me, WDYT @sebastiansturm ?

[sebastiansturm]

hmm, thanks for the heads-up, that seems to make sense (to some degree). Looks to me like font-lock-fontified is always t if font-lock-support-mode is set to jit-lock-mode, I guess in your case it isn't @gitonthescene? I probably have to change something then (perhaps use font-lock-ensure, but at least in Emacs 29, font-lock-ensure-function seems to be a no-op if font-lock-fontified is t, so I'd guess by just using that instead I'd break the jit-lock-mode case?

Perhaps I should just call font-lock-after-change-function, though it's not clear to me whether that one is supposed to be part of font-lock's external API.

@gitonthescene: could you please check whether jit-lock-mode is enabled in your config and whether font-lock-after-change-function instead of font-lock-flush works for you?

[gitonthescene]

@sebastiansturm - font-lock-support-mode is set to jit-lock-mode even while font-lock-fontified is nil. Where in the code do you see a connection between these settings?

As I said, I made a barebones setup to start working on support for an esolang, so I’m probably not requireing something 99% of other setups do.

[gitonthescene]

I see I missed the last question. font-lock-flush does work for me and is what lead me to figure out the issue.

[sebastiansturm]

@gitonthescene sorry for not getting back to you sooner, glad you could figure it out already! Should we add something to the readme, in case others run into the same issue?

[gitonthescene]

@sebastiansturm - Sorry. Nothing has changed since my first post. I’m simply answering your last question. It’s really a question of how you want to handle this. I think changing to font-lock-ensure would work. I also think it might make sense to set font-lock-fontified when triggering the callback. Or maybe after? You won’t get fontification the first time the hook gets called with this last but it makes logical sense for it to be there.

[sebastiansturm]

@gitonthescene oh, sorry, I misunderstood you then. Though I'm a bit confused -- when you're saying "font-lock-flush does work for me", did you mean to refer to font-lock-ensure or to font-lock-after-change-function? I thought font-lock-flush was just the thing that didn't work for you? As for your previous question, when font-lock-mode activates, it should call font-lock-mode-internal (via font-lock-default-function), which would call font-lock-turn-on-thing-lock, which in your case (since jit-lock-mode is enabled) should set font-lock-fontified to t. Are you using font-lock mode at all? Does debug-on-entry (applied to font-lock-mode-internal, for example) trigger when you're opening a fresh buffer (where you'd expect lsp semantic highlighting to work)?

[gitonthescene]

Sorry, I did mean font-look-ensure. I edited the comment above but that probably didn’t come through in the update notification.

As I say I’m developing this from scratch so I might not be following some conventions. I was not explicitly using font-lock-mode anywhere on the theory it was being provided by lsp-mode. Just now I tested though and running font-lock mode after opening the buffer using lsp-mode didn’t help. Similarly adding a #'font-lock-mode hook for my mode (ngnk-mode) either before or after the #'lsp doesn’t help.

Also font-lock-mode-internal does not trigger.

[gitonthescene]

@sebastiansturm Are you able to reproduce any of the reported results in any sort of a test environment?

If it helps, I've attached my bare-bones mode definition. ngnk-mode.el.txt

[sebastiansturm]

@gitonthescene no, I don't see these issues with the modes I regularly use. How do I set up your language server for ngnk-mode? I assume I have to install pygls, and then clone some extra repository? Can you provide a link, and ideally, a matching lsp-mode config? thanks!

[gitonthescene]

I mocked up a dummy repository. Maybe this could be used as the basis for some unit tests or something.

Let me know if you have questions. Also, would it make sense to turn this into a separate issue?

[sebastiansturm]

@gitonthescene thanks, that is an excellent reproducer! It does work for me though, see the attached screenshot.

These are the modes enabled in my dummy-mode buffer after following your setup instructions:

Minor modes enabled in this buffer: +Emacs-Lisp-Non-Package +Modeline +Word-Wrap Apheleia Auto-Save
Better-Jumper-Local Corfu Diff-Hl Display-Line-Numbers Dtrt-Indent Evil-Local Flymake Flymake-Popon
Font-Lock Hl-Line Lsp-Diagnostics Lsp-Managed Lsp Lsp-Modeline-Diagnostics
Lsp-Modeline-Workspace-Status Lsp-Semantic-Tokens Visual-Line Whitespace Ws-Butler Yas

The major mode is DummyLSP mode defined in /tmp/dummy-lsp-mode/dummy-mode.el```

[gitonthescene]

I can only guess the difference is something in your init file. Next step is to run this with as empty an init file as possible.

[gitonthescene]

@sebastiansturm In any event it sounds like you're not interested in investing any energy into this. I'll leave you to do with this what you want.

[sebastiansturm]

@gitonthescene that's not the case, though I understand you must be frustrated with the slow progress on this issue.

With a completely unconfigured Emacs instance, I have now been able to reproduce your issue, and I think it's because font-lock considers itself unconfigured. If you'll have a look at font-lock.el, in particular font-lock-specified-p, there's a check for font-lock-defaults or font-lock-keywords being defined (which the example dummy mode doesn't do). So I assume that if you're planning to define some basic fontification properties on top of which semantic fontification would apply, things would just work. If that is not the case, and you'd rather like to leave fontification entirely to lsp, then I'm not yet sure how to best resolve this. Since semantic fontification is supposed to work on top of font-lock, it seems to me that font-lock should be brought into a state where it considers itself operational. Doing something like (font-lock-mode-internal t) within your major mode definition does define defaults, and hence "fixes" semantic fontification too, but calling something from user code that ends in -internal is probably a bad idea. So, I'll have to browse font-lock documentation a bit and see if there's a recommended proper way to do this, but again this will take some time as I'll get to it no sooner than tomorrow evening (at best).

Edit: instead of (font-lock-mode-internal t), your major mode could call (font-lock-set-defaults) I think; alternatively, I might want to call that within lsp--semantic-tokens-initialize-buffer; that should be acceptable too if there are no cases where this causes trouble by, e.g., overwriting user-defined fontification rules

[gitonthescene]

@sebastiansturm - Glad you were able reproduce the issue.