[Feature Request] syntax coloring of Python f-strings by default

[peterjc]

Description of the problem or steps to reproduce

Current behavior of micro is to treat Python f-strings like plain strings (yellow in the default color scheme):

This example is taken from https://docs.python.org/3/reference/lexical_analysis.html#f-strings where the variables like name in the first f-string of this example are in white (screenshot with Firefox showing coloring of the expressions with f-strings):

Desired behaviour would be like the Python.org documentation, or similar - this is Sublime Text Build 4189 on macOS which seems to format the code snippets within f-strings like normal Python code:

Specifications

$ micro --version
Version: 2.0.14
Commit hash: 04c57704
Compiled on August 27, 2024

Commit hash: 04c57704 OS: Linux Terminal: Ghostty

(Same behavior on macOS with micro 2.0.14)

[Andriamanitra]

Unfortunately the recursive nature of the f-string syntax makes this impossible to solve properly with micro's current syntax highlighting system. I actually use a custom Python syntax file that tries to highlights f-strings, but it's so buggy that I don't think it would be a good idea to have it built into the editor.

https://github.com/Andriamanitra/dotfiles/blob/c642702c048e7a045fb08f7e5792e981b41c07b7/micro/.config/micro/syntax/python3.yaml#L76-L136

[Andriamanitra]

And here is an attempt using the "one level of recursion" trick from #2596. Still broken for the nested fields, and I can't figure out a way to make everything after : the same color. (and for some reason the # is broken too, that might just be a bug in micro).

https://gist.github.com/Andriamanitra/7dc25fc510c0fee4e54883cd28d42487

[peterjc]

The full syntax is understandably very tricky - perhaps a minimal solution of highlighting anything matching {...} within an f-string a different color would be a safe modest improvement?

[Andriamanitra]

The full syntax is understandably very tricky - perhaps a minimal solution of highlighting anything matching {...} within an f-string a different color would be a safe modest improvement?

This handles that up to one level of nesting, although I'm a bit worried about the regexp performance with how many wild cards the pattern is using. It likely blows up if you give it a specifically crafted string.

    - constant.string:
        start: "f\""
        end: "\""
        skip: "\\\\."
        rules:
            - constant.specialChar: "\\\\['\"nrtbf\\\\]"
            - constant.specialChar: "\\\\0[0-7]{0,2}"
            - constant.specialChar: "\\\\x[0-9A-Fa-f]{2}"
            - constant.specialChar: "\\\\u[0-9A-Fa-f]{4}"
            - constant.specialChar: "\\\\U[0-9A-Fa-f]{8}"
            # hard-coded cyan because I'm not sure which group would fit well
            - cyan: "[{](([{][^}]*[}])?[^{}]*)*[}]"
            - symbol.brackets: "[{}]"

[JoeKar]

And here is an attempt using the "one level of recursion" trick from #2596.

I think it would be more straight forward to allow recursion to a certain level. Sure, the parsing and highlighting system needs some additional tuning, but it isn't impossible.

This handles that up to one level of nesting, although I'm a bit worried about the regexp performance with how many wild cards the pattern is using. It likely blows up if you give it a specifically crafted string.

Yep, the current released highlighter isn't prepared for such stuff and already fails on "simple" html and xml. So definitely not before #3127.

[Andriamanitra]

And here is an attempt using the "one level of recursion" trick from #2596.

I think it would be more straight forward to allow recursion to a certain level. Sure, the parsing and highlighting system needs some additional tuning, but it isn't impossible.

Do you mean that it isn't impossible with the current system, or that it isn't impossible to change the system to make it possible? I think the parser would need some way to maintain state, as it's not possible to (ab)use the start/end/skip/rules system for nested expressions like this:

w = 7
f"{ {"width":f"{w:{w}}"} !r:{w}}"

[JoeKar]

[...] or that it isn't impossible to change the system to make it possible?

Yep. The parser already maintains states. In the version with #3127 it will have the capability to remove already identified regions in case of overlaps.

[...] as it's not possible to (ab)use the start/end/skip/rules system for nested expressions like this:

Yes, it will require more features, but I'm unsure if this weird example can be solved easily. :sweat_smile:

[peterjc]

I think the simple one level of recursion example shown would be a worthwhile incremental improvement (e.g. it should help visually flag unmatched { and }).

[usfbih8u]

Is this the same problem that happens in markdown syntax?

**For example, this phrase loses the bold syntax after using `backticks`**.

> Similar happens here after using `backticks` this text is not highligthed

[JoeKar]

Is this the same problem that happens in markdown syntax?

**For example, this phrase loses the bold syntax after using `backticks`**.

> Similar happens here after using `backticks` this text is not highligthed

Actual:

https://github.com/zyedidia/micro/pull/3127:

[usfbih8u]

#3127:

Great! Thank you.