LaTeX syntax highlighting in Rmd

[gadenbuie]

• VSCode Version: 1.28.0
• VSCode-R Version: v0.6.2
• OS Version: x86_64-apple-darwin15.6.0 (64-bit)

Steps to Reproduce:

1. Use two reserved markdown characters in LaTeX math blocks in the same line
2. Syntax highlighting does not consider the LaTeX math notation and applies styling

Example of issue (with totally made up math)

Expected something more like this

TBH, simply catching the use of LaTeX and removing the syntax highlighting between paired $..$ or $$..$$ would be a great improvement.

Example text

For each $x_i$, consider all $x_i < k$.
Is $x*2$ greater than $y*2$?

$$
y_i = m * x_i + b
$$

[andycraig]

Once PR #228 is merged, I'll fix this so that text in $..$ and $$..$$ is highlighted as LaTeX (if the user has installed an extension that provides LaTeX highlighting), or has no highlighting (if they haven't installed such an extension).

[storopoli]

Guys do we have any news on LaTeX syntax highlighting?

[andycraig]

Hi @storopoli, I was planning on working on this around this time last year but it fell too far down my list of priorities.

The file that needs editing is https://github.com/Ikuyadeu/vscode-R/blob/master/syntax/Markdown%20Redcarpet.json . $$..$$ blocks will have a similar format to the fenced_blocks, and $..$ will be similar to code-inline-r. Provided the user has an extension like this installed, I think this should work for the patterns field:

"patterns" : [
    {"include" : "source.latex"}
]

[storopoli]

maybe something like this:

"inline" : {
            "patterns" : [
                {"include" : "source.latex"}
            ],
            "repository" : {
                "code-inline-latex" : {
                    "begin" : "([$|$$][ ]+)",
                    "beginCaptures" : {
                        "1" : {
                            "name" : "punctuation.definition.raw.rmarkdown"
                        }
                    },
                    "end" : "([$|$$])",
                    "endCaptures" : {
                        "1" : {
                            "name" : "punctuation.definition.raw.rmarkdown"
                        }
                    }
                }

[andycraig]

@storopoli That looks like the right sort of idea. I think it would match $$..$ and $..$$ though.

A PR for this feature would be very welcome! :)

[storopoli]

Ok, I'm new to VS Code Extensions, but I wanna help. How do I make LaTeX Workshop render stuff from .Rmd files? Can you point me towards where I can find how to do this?

I could try to implement a simple solution with just $....$ and $$...$$ math preview. Then we could analyze how to make it work with more advanced LaTeX stuff like begin{aligned|cases}...\end{aligned|cases} and \begin{matrix|bmatrix|pmatrix}...\end{matrix|bmatrix|pmatrix} etc.

[andycraig]

Great!

We use embedded languages to provide syntax highlighting for code blocks of Python, C etc. in Rmd files. It should be possible to do this for LaTeX too. The difference is that grammars for Python, C etc. are built in to VS Code itself, but LaTeX isn't built in. (The built-in languages are here: https://github.com/microsoft/vscode/tree/master/extensions)

LaTeX Workshop defines a LaTeX grammar: https://github.com/James-Yu/LaTeX-Workshop/blob/master/package.json#L172 So my (untested!) theory is that if LaTeX is added as an embedded language to the Rmd syntax file, and the user has installed LaTeX Workshop, then highlighting for begin{aligned|cases}...\end{aligned|cases} etc. should happen automatically. It might also be necessary to activate the extension by, e.g., opening a .tex file.

Here are the changes I think are necessary:

1. Modify https://github.com/Ikuyadeu/vscode-R/blob/master/syntax/Markdown%20Redcarpet.json with something like what you posted above. I think the top should be like this:

"inline" : {
            "patterns" : [
                {"include" : "#code-inline-latex"}
            ],
            "repository" : {
                "code-inline-latex" : {

Also you'll need something like this:

                    "contentName" : "meta.embedded.block.latex",
                    "patterns" : [
                        {"include" : "source.latex"}
                    ]

2. Add a new line to this section, for meta.embedded.block.latex: https://github.com/Ikuyadeu/vscode-R/blob/master/package.json#L244-L249

The command Developer: Inspect Editor Tokens and Scopes is very helpful for checking what VS Code thinks the syntax and highlighting are.

Hope that helps! We appreciate contributions so if things are still unclear or you get stuck etc. please do ask and I'll try to provide more info.

[storopoli]

I've made the changes in #553

Unfortunately I don't know how to test the extension. I've trying running with F5 but to no avail.

[andycraig]

Excellent! For testing: File menu -> 'Open Folder...' and choose select 'vscode-R' (the directory with your edits). Then press F5. This should open a new VS Code window which is running the modified version of vscode-R with your changes. If F5 doesn't work, could you let me know what happens (error message etc.)? Thank you.

[storopoli]

Ok. It doesn't render LaTeX for a given .Rmd file. Here is an example for a tutorial in Portuguese that I am developing for a Bayesian Statistics Course

[andycraig]

There's an error showing at the bottom right. My guess is that you need to build the extension. Here are some instructions: https://github.com/Ikuyadeu/vscode-R/wiki/Contributing

[storopoli]

I've managed to build it without errors. But unfortunately LaTeX doesn't render. How do I force LaTeX Workshop extension to be loaded with vscode-R in the F5 run?

[andycraig]

Does the window shown in the screenshot show '[Extension Development Host]' in the title bar? If so, good - it's running your edited version of vscode-R. If not, it's running the release version of vscode-R.

Note that the 'textmate scopes' section shows just

meta.paragraph.markdown
text.html.rmarkdown

It should show meta.embedded.block.latex. Since it doesn't show that, it probably means that something isn't right in your code-inline-latex code and so VS Code isn't recognising $$..$$ as matching code-inline-latex. The next step is to figure out why. I suggest using an extremely basic R Markdown file, for example just

`r 1` $$ 2 $$

and making minor changes to the code-inline-latex section until it can successfully identify the $$ 2 $$ as meta.embedded.block.latex. This can be a bit tricky because VS Code won't usually throw errors if the syntax definition is wrong, it will just fail silently, so some trial and error might be involved.

One possibility is that the $ in the syntax file need escaping: \$. I'm not sure but it's something to try.

[storopoli]

Ok some insights (yes, the extension loaded just OK in the title bar [Extension Development Host].

1 - I've went to the Redcarpet repository to find something related to latex and found this issue Support for latex equations, it just hints to treat latex as a language code block.

2- I've tried changing stuff in the code-inline-latex and also using escaped \$ inside the .Rmd, but to no avail. See image below

3- I've tried inserting a latex fenced_block inside Markdown Redcarpet.json to figure what it was rendering:

"fenced_block_latex" : {
                    "begin" : "(^|\\G)([`]{3})\\{*[ ]*(?:latex).*\\}*$",
                    "end" : "(^|\\G)([`]{3})($|\\z)",
                    "name" : "meta.embedded.block.latex",
                    "patterns" : [
                        {"include" : "source.latex"}
                    ]
                }

and I've discovered that it is not rendering as LaTeX but as rmd, see screenshots below comparing with the js block:

[andycraig]

Some suggestions:

2. The \$ is for use in the syntax file, not the Rmd file. Also try \\$ in the syntax file.
3. That's a good experiment. It looks like your definition of fenced_block_latex uses {latex}, so try this in the Rmd file:

    ```{latex}
    x = 1
    ```

[storopoli]

Doesn't work also.

I was diving into some details. If you see the L339 in README.md you will find:

* Extended Syntax(R, R Markdown, R Documentation)
--
… |  
338 | * [r.tmbundle](https://github.com/textmate/r.tmbundle)
339 | * [markdown-redcarpet.tmbundle](https://github.com/streeter/markdown-redcarpet.tmbundle)

So I went to the streeter's rmarkdown-redcarpet.tmbundle repository. It's an old archived repository and there is a proposal from 2016 that instructs on how to render LaTeX using Mathjax. Could that be helpful to us?

[andycraig]

None of those suggestions worked? If my suggestion for 3. didn't work it makes me wonder if any of the changes are actually having an effect. Can you make some major change and check that it has an effect? For example, try deleting all the fenced blocks, and then check the textmate scopes for this:

    ```{r}
    x <- 1
    ```

If it still shows as meta.embedded.block.r then it means something is wrong and your changes aren't having any effect.

I don't think that Mathjax proposal is related to this because we're trying to get LaTeX syntax highlighting rather than render LaTeX.

[storopoli]

You are right, the changes are not having any effect at all.

With or without the fenced blocks in the file Markdown Redcarpet.json it shows the same in both:

[andycraig]

That's what I would hope to see without the fenced blocks in the syntax file. It would mean that your changes were having an effect, and removing the fenced blocks stopped x <- 1 from being highlighted as R.

But I would not expect to see the same thing WITH the fenced blocks in the syntax file. With the fenced blocks in the syntax file I would expect to see textmate scopes like in this screenshot from my PC running vscode-R 1.6.4:

Can you post a screenshot of that Rmd file again, WITH the fenced blocks in the syntax file, but this time showing the whole VS Code window? I'm wondering if there might be some clues in the title or in the status bar.

[storopoli]

Here you go

[andycraig]

I've added a couple of comments to your PR. There was a } in the wrong place, so the syntax file wouldn't have been loading properly. That would explain why the textmate scopes weren't what I was expecting, and why your changes weren't having any effect.

[storopoli]

I've corrected the brackets { in the PR now the rmd r chunks and inline r code shows the correct textmate scopes expected. But I still get the same behavior from $ and $$ inline:

[andycraig]

I think it's best to approach this as two tasks:

1. Detect $$..$$ blocks. I suggest using source.r instead of source.latex for now, since we know source.r works. Try to modify the syntax file so that text in $$..$$ blocks is highlighted as R code. I'm 90% sure that you'll need \\$
2. Get $$..$$ blocks highlighting as LaTeX code

Number 2 is more difficult. It may take some more research and a lot of trial and error. The current languages supported in R Markdown files are all built in to VS Code. LaTeX is not built in to VS Code. I have no experience adding support for a language not built in to VS Code.

Ideas for number 2:

• Look at the syntax files for the LaTeX Workshop extension: there might be some clues
• Check syntax files for some other LaTeX extensions (LaTeX Workshop is the most popular one but there are many others)
• Consider that source.latex may not be correct
• Consider that you may have to modify package.json more
• When testing, make sure that an extension that provides LaTeX grammar (e.g., LaTeX Workshop) is activated

[storopoli]

You are definitely right:

1. I did need \\$
2. There is something wrong with source.latex see the pic with source.r instead of source.latex.

[andycraig]

Great! That's task 1 finished. Next is task 2. Like I said, it will probably be more difficult and involve a lot of trial and error. Hopefully the ideas I posted above help.

[eitsupi]

We use embedded languages to provide syntax highlighting for code blocks of Python, C etc. in Rmd files. It should be possible to do this for LaTeX too. The difference is that grammars for Python, C etc. are built in to VS Code itself, but LaTeX isn't built in. (The built-in languages are here: microsoft/vscode@master/extensions)

FYI, LaTeX is now built in code-insiders.

[andycraig]

Thanks @eitsupi! That should make this much easier.

[eitsupi]

As I posted in https://github.com/REditorSupport/vscode-R/pull/553#issuecomment-1030550235, I can't seem to capture the end $ because the latex syntax contains $. It looks like we need to have a separate latex definition for R Markdown.

If we experiment with adding new definitions to the code chunk, we will find that even the last ``` is interpreted as a latex and cannot be captured.

[MatrixRanger98]

Any update on this? This feature is indeed needed if you rely heavily on R Markdown. Also, I believe using embedded language is NOT the right way to implement this. Instead, we should borrow ideas from the built-in extension, vscode.markdown-math.

[MatrixRanger98]

As VSCode has a built-in syntax highlighting for Markdown math, it is possible to reuse the same syntax for R Markdown math.