Incorrect syntax coloring for formatted-raw string literals (example: fr"{some_path_here}\some_file_here.txt")

[matthewgdv]

trafficstars

Hi,

I wanted to draw attention to the fact that formatted-raw string literals are syntax highlighted by MagicPython as if they were ordinary raw string literals, without any special syntax highlighting given to the contents of the curly braces. See examples below:

Is there any chance this could fixed/enhanced?

Cheers!

[vpetrovykh]

Compare the following:

# plain string
work_folder_dir  = "{current_dir}\WorkFolders"
# r - regexp highlighted (that's why escaped 
#     '\W' is still special there, but '{...}' is not)
work_folder_dir  = r"{current_dir}\WorkFolders"
# R - raw string
work_folder_dir  = R"{current_dir}\WorkFolders"
# f - plain format string
work_folder_dir  = f"{current_dir}\WorkFolders"

# this is more like 'r' - regexp, so the format is ignored in favor of regexp
work_folder_dir  = fr"{current_dir}\WorkFolders"
# this is like 'R' - raw + format
work_folder_dir  = fR"{current_dir}\WorkFolders"

On the one hand, it's pretty clear that whatever color scheme you're using is not taking advantage of all of the nuances of string highlighting (such as showing escaped characters differently). On the other hand, there's an inherent problem with the dichotomy between pure Python in which raw strings are just raw strings and the reality that typically it's raw strings that get used for non-trivial regular expressions. The decision to highlight r strings as regexp and R strings as just raw is explained in this issue: #114.

Personally, I'd suggest using fR or FR instead of fr if you can, because that will get you what you're asking for.

[matthewgdv]

Wow, thanks a bunch @vpetrovykh, I had no idea there was a difference between r"" and R"" raw strings. Turns out there was no issue here at all, I'm happy to close it. Thanks for taking the time to answer!

[vpetrovykh]

@Mettpawwz to be clear, there is no difference between r and R strings in Python, but historically some editors/highlighters treated r-strings as regexp as a matter of convention.

[vors]

This "feature" is super confusing and annoying, please consider dropping it. It causes a lot of confusion which is evident from the linked bugs. I don't think that "this is how it was done in some old implementation of syntax highlighter" is a compelling argument to keep it.

[matthewgdv]

First off:

@vors I disagree. Both forms of raw strings have important use-cases, and removing either one as an option reduces user agency for no benefit. Unless you want to implement some hacky workaround for making the grammar aware of user intent like special nightmarish MagicPython-specific comments (how would this even work with multiple strings on one line?) then this is the best way to do it.

This behaviour is documented in MagicPython's README.md, so a little googling should reveal the way this works to anyone who's new to the grammar.

Secondly, I'm reopening this because there actually is an issue here, although not the one I originally thought:

disclaimer: I've looked through the list of MagicPython scopes and couldn't find any that seemed to do what I'm going to describe, but that doesn't mean they don't exist. There are some scopes where I can't work out from the names exactly what they do so if I'm wrong and this is already supported then disregard this and please let me know what scopes I should be using in my syntax theme. I will then close this issue again.

Currently the grammar doesn't seem to be able to handle f-string braces within fr-string patterns the way it does in normal f-strings. For example:

In my syntax theme, operators and numerics are light blue and f-string braces are orange and you can see that in the first example. In the fr-string though that same interpolation just shows as the default regex color for both the braces and the contents. The good news is that at least it does syntax highlight the escaped regex quantifier braces correctly (dark green).

The syntax theme I'm using is my own, you can look at it for an explanation of the colour scheme:

https://atom.io/themes/dark-snek-syntax

Is there a scope I could be taking advantage of to get the proper f-string syntax highlighting for fr-strings, or is this something that MagicPython doesn't currently support? If so, I'd like to raise it as an enhancement.