Positron should better recognize and highlight Python keywords

[ntluong95]

Currently, module keywords such as pandas, pd, etc. are not highlighted. See the example below

In Vscode

In Positron

[petetronic]

Thanks for logging this, this is as a result of the token types from of the Jedi-based language server in use in Positron's Python LSP.

[ntluong95]

Is there anyway to use Pylang, which is much better?

[seeM]

@ntluong95 I can't find any info on a "Pylang" syntax highlighter. Do you mean Pylance?

[ntluong95]

Yes sorry I meant Pylance. Fortunately I found this one, which serve my need. Maybe something you can consider to integrate into Positron as I know Pylance is only for VSCode. Link: https://open-vsx.org/extension/detachhead/basedpyright

[ntluong95]

Yes sorry I meant Pylance. Fortunately I found this one, which serve my need. Maybe something you can consider to integrate into Positron as I know Pylance is only for VSCode. Link: https://open-vsx.org/extension/detachhead/basedpyright

Here is the result after installing the package. However, the Quarto file could not display syntax highlighting for Python module. I think Positron should have more support for Quarto extension since now the Jupyter Notebook is far more better

[seeM]

Thank you for pointing out basedpyright, we'll look into it!

Aside from the Quarto issue you mentioned, how well does basedpyright currently work in Positron? At the very least, we may be able to iron some of those issues so that users can choose alternative language servers – although it is worth noting that completions would no longer be aware of your session's variables and would not benefit from other Positron-specific language server features in future.

[ntluong95]

It works well, excepted that in the Outlines, you will see 2 language servers. I think if you can look at their code and cherry-pick the part that help to recognize Python module keywords, and also attach it to Quarto, the problem will be solved. Just my naive idea

[seeM]

Could you please share a screenshot of the 2 language servers in Outlines?

[ntluong95]

Attached is the screenshot

[ntluong95]

But is there any plan to support display syntax highlighting of Python code for Quarto too?

[juliasilge]

We do have syntax highlighting for Python code in Quarto, yes:

Is there a specific syntax highlighting feature you are missing or problem you are observing?

[ntluong95]

Currently, module keywords such as pandas, pd, etc. are not highlighted. See the example below

In Vscode

In Positron

I think module keywords, and semantic highlighting from Pylance is missing. That's why I open this thread

[KJHards]

+1 for this change. Tried to switch from VScode but too much of my python code has no syntax highlighting

[lionel-]

The ruff team is "considering" semantic highlighting "but not in the near future" (see https://github.com/astral-sh/ruff-vscode/issues/438#issuecomment-2033721488). Tracking issue: https://github.com/astral-sh/ruff-lsp/issues/237

[Giorgione1609]

+1 for this change. Tried to switch from VScode but too much of my python code has no syntax highlighting

Same here. Are there any news? It's such a pity to drop Positron because of this.

[seeM]

Unfortunately this isn't a priority at the moment, but I did spend a little time seeing what it would take to support semantic highlighting with Jedi on the 3731-investigate-jedi-lsp-semantic-tokens branch if anyone is interested in pursuing this further.

That should get basic semantic highlighting. From my understanding, it's easy with Jedi to classify declarations (e.g. in def f(): pass, f is a declaration of a function). The trickier part will be correctly classifying references (e.g. calling f later in your code), especially references to objects in larger external modules. I think the correct Jedi API is infer() which might be too slow for the semantic highlighting usecase.

The language server protocol also supports updating semantic tokens given edits to the document (rather than starting from scratch each time). I haven't looked into whether that's necessary (or possible) with Jedi. Looking at this comment, Jedi's parser parso heavily utilizes caching of different parts of the document for performance, so it might not be necessary to support semantic token edits.

It would be helpful to know from you all which token types are the most important, and we can look into the feasibility of getting that to work with Jedi.

[ntluong95]

I don't have the knowledge on this, but this document might be helpful to know all of the token type.

For prioritizing purpose, for me at least module keywords, class methods, function parameters, This one is not so good, but have some key takeaways that we can start from I hope

[lbeiby]

@seeM how is the progress. I think Jedi is quite good in general but hope Positron can implement something like customized Jedi like Microsoft did with Pylance, which improve a lot of UX

[ntluong95]

I tested and it works quite well. May be add also "variable.declaration", "property.declaration" and "function.builtin". Those are something I have set syntax highlighting when using Pylance.

The problem is it doesn't work on Jupyter notebook. But Pylance doesn't work on Quarto document. I think the current highlighting is good enough for me at least to move forward, as long as Jupyter Notebook and Quarto document support for it, I may not need Pylance. I can use Ruff for type checking and linting

[ntluong95]

I found this one, could be helpful: https://gitee.com/guanyusheng/semshi/blob/master/rplugin/python3/semshi/parser.py

[juliasilge]

In https://github.com/posit-dev/positron/discussions/7328 we have a request for type inlay hints, like what Pylance offers.

[ntluong95]

Unfortunately this isn't a priority at the moment, but I did spend a little time seeing what it would take to support semantic highlighting with Jedi on the 3731-investigate-jedi-lsp-semantic-tokens branch if anyone is interested in pursuing this further.

That should get basic semantic highlighting. From my understanding, it's easy with Jedi to classify declarations (e.g. in def f(): pass, f is a declaration of a function). The trickier part will be correctly classifying references (e.g. calling f later in your code), especially references to objects in larger external modules. I think the correct Jedi API is infer() which might be too slow for the semantic highlighting usecase.

The language server protocol also supports updating semantic tokens given edits to the document (rather than starting from scratch each time). I haven't looked into whether that's necessary (or possible) with Jedi. Looking at this comment, Jedi's parser parso heavily utilizes caching of different parts of the document for performance, so it might not be necessary to support semantic token edits.

It would be helpful to know from you all which token types are the most important, and we can look into the feasibility of getting that to work with Jedi.

Can you implement a minimal of semantic highlighting at least? I have a look on your commit and it seems quite simple to merge now @seeM

[seeM]

This has been implemented in jedi-language-server but still seems to have a bug (https://github.com/pappasam/jedi-language-server/issues/340). We could wait for that to be resolved.

[austin3dickey]

The ecosystem for Python language server extensions has grown considerably this year. I've noticed some users and developers have been trying these in Positron, though I can't vouch for the full experience yet.

• https://open-vsx.org/extension/astral-sh/ty
• https://open-vsx.org/extension/detachhead/basedpyright
  • watch out for https://github.com/posit-dev/positron/issues/7393
• https://open-vsx.org/extension/zuban/zubanls
• https://open-vsx.org/extension/meta/pyrefly

We are still planning to improve the built-in experience with access to Positron kernel information, but trying one or more of these may be beneficial to people looking for additional features today.

[ntluong95]

@austin3dickey for semantic token provider, see some implementations in ty here: https://github.com/astral-sh/ty/issues/791

[ntluong95]

Pyrefly has the semantic highlight now. Tested and worked well in Positron

[austin3dickey]

I'm scoping this issue to Python files and notebooks, and will track semantic token support in Quarto files in https://github.com/quarto-dev/quarto/issues/420.

We will bootstrap pyrefly, which adds better semantic token highlighting to Python files and notebooks out of the box.