Add `.hidden` block feature like in Quarto to support `\newcommand` for Mathjax without breaking PDF

[jassler]

While custom macros using \newcommand{} work fine in the rendered output, RStudio does not seem to be able to display them correctly.

Take the following simple example.

---
title: "My maths"
output: html_document
---

\newcommand{\e}{\mathcal{E}}

$$
\e
$$

The output in my html and pdf document is fine, but in RStudio the editor pane looks like this.

I hope it is possible to somehow have the \e macro be evaluated before it is rendered. For consistency in my documents, I like to use a lot of macros the same way I would use constant variables or enums in programming. But the preview becomes unusable this way.

session_info

R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin22.1.0 (64-bit)
Running under: macOS Ventura 13.2.1, RStudio 2022.12.0.353

Locale: en_US.UTF-8 / en_US.UTF-8 / en_US.UTF-8 / C / en_US.UTF-8 / en_US.UTF-8

Package version:
  base64enc_0.1.3 bslib_0.4.2     cachem_1.0.7    cli_3.6.0       digest_0.6.31   ellipsis_0.3.2  evaluate_0.20   fastmap_1.1.1  
  fs_1.6.1        glue_1.6.2      graphics_4.2.2  grDevices_4.2.2 highr_0.10      htmltools_0.5.4 jquerylib_0.1.4 jsonlite_1.8.4 
  knitr_1.42      lifecycle_1.0.3 magrittr_2.0.3  memoise_2.0.1   methods_4.2.2   mime_0.12       R6_2.5.1        rappdirs_0.3.3 
  rlang_1.0.6     rmarkdown_2.20  sass_0.4.5      stats_4.2.2     stringi_1.7.12  stringr_1.5.0   tinytex_0.44    tools_4.2.2    
  utils_4.2.2     vctrs_0.5.2     xfun_0.37       yaml_2.3.7     

Pandoc version: 3.1

Checklist

When filing a bug report, please check the boxes below to confirm that you have provided us with the information we need. Have you:

• [x] formatted your issue so it is easier for us to read?

• [x] included a minimal, self-contained, and reproducible example?

• [x] pasted the output from xfun::session_info('rmarkdown') in your issue?

• [x] upgraded all your packages to their latest versions (including your versions of R, the RStudio IDE, and relevant R packages)?

• [ ] installed and tested your bug with the development version of the rmarkdown package using remotes::install_github("rstudio/rmarkdown")?

[cderv]

Putting the command inside the display math will correctly adapt the preview.

Is there a reason you need to put it outside ?

It needs to be inside a math environment for it to be taken into account with Mathjax

[jassler]

Oh, I didn't know that, thanks! I encountered someone with the same problem on StackOverflow and couldn't find other places discussing this.

(feel free to add your answer to the StackOverflow question)

[jassler]

Just a small heads-up that this breaks for tex / pdf (I assume that \newcommand is scoped to the math environment).

As a workaround, I came up with the following solution that breaks out of the math environment if the output is not an html.

---
title: "My maths"
output: pdf_document
---

$$`r if(!knitr::is_html_output()) ' \\]'`
\newcommand{\e}{\mathcal{E}}
`r if(!knitr::is_html_output()) '\\[ '`$$

$$
\e
$$

I've edited your answer on StackOverflow, I hope it's alright.

[cderv]

Just a small heads-up that this breaks for tex / pdf (I assume that \newcommand is scoped to the math environment).

Oh I see I think - to be sure to understand you mean that you want to define a command once, and use it in several math part, and you want the preview to work. Is that right ?

For now if you do

\newcommand{\e}{\mathcal{E}}

$$
\e
$$

it works in PDF

if you do

$$
\newcommand{\e}{\mathcal{E}}
\e
$$

it works in HTML PDF and PREVIEW

but if you do

$$
\newcommand{\e}{\mathcal{E}}
$$

$$
\e
$$

it works in HTML and PREVIEW but not in PDF.

Is this correct ?

[cderv]

BTW the cleaner solution so support both LaTeX and HTML could be this one. But it breaks the preview in IDE.

---
title: test
output: 
  html_document: default
  pdf_document:
    keep_tex: true
header-includes:
  - |
    ```{=latex}
    \newcommand{\e}{\mathcal{E}}
    ```
---

```{=html}
$$
\newcommand{\e}{\mathcal{E}}
$$
```

Define the command in the first math display where you need to use the command 

$$
\e
$$

And then in other math display, it will still work 

$$
\underline{\e}
$$

Your solution has the advantage to support both (I would use knitr::is_latex_output() though) and not break the preview.

However, it is a bit verbose to write - probably doing a RStudio snippet would help inserting this.

I am thinking we can maybe improve this with a special syntax using Pandoc's fenced div. This would probably work for all 3 cases

[jassler]

you want to define a command once, and use it in several math part, and you want the preview to work

Yeah. Or also just produce pdfs with the advantage of previewing the equations in RStudio. That's how I got to my verbose answer, to make sure that RStudio knows how to interpret it and still produce valid tex code.

And you are correct, doing something like this would look cleaner (and seems to cause fewer issues in the IDE)

---
title: test
output: 
  html_document: default
  pdf_document:
    keep_tex: true
---

$$
\newcommand{\e}{\mathbb{E}}
$$

````{=latex}
\newcommand{\e}{\mathbb{E}}
````

$$
\e
$$

I noticed however that LaTeX still parses \newcommand in math mode. So things like \newcommand{\bm}[1]{\boldsymbol{#1}} causes LaTeX to quit and complain about Command \bm already defined.

Conditionally commenting this import out with `r if(!knitr::is_html_output()) ' % '` \newcommand{... would solve this again, but it becomes way more verbose again. And, as you've mentioned, using ````{=html} works, but doesn't get interpreted by RStudio.

Do you know another way to solve this? Maybe there exists a preprocessor step?

All in all however, I have found a way that works for me. So if there's no real easier way, this is ok for me.

[cderv]

I am thinking we can maybe improve this with a special syntax using Pandoc's fenced div. This would probably work for all 3 cases

Here is a prototype for this, inspired by more advanced feature in Quarto (https://quarto.org/docs/authoring/conditional.html)

---
title: "My maths"
output: 
  pdf_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("when-format.lua")
  html_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("when-format.lua")
header-includes:
  - |
    ```{=latex}
    \newcommand{\e}{\mathcal{E}}
    ```
---

```{cat}
#| engine.opts:
#|    file: when-format.lua
#| echo: false
Div = function(div)
  if div.classes:includes("content-visible") then
    local cond = div.attributes["when-format"]
    local format = FORMAT
    if string.sub(format, 1, 4) == "html" then format = "html" end
    if div.attributes["when-format"] == format then
      return div.content
    else
      return {}
    end
  end
end
```


::: {.content-visible when-format="html"}
$$
\newcommand{\e}{\mathcal{E}}
$$
:::

$$
\e
$$

I noticed however that LaTeX still parses \newcommand in math mode. So things like \newcommand{\bm}[1]{\boldsymbol{#1}} causes LaTeX to quit and complain about Command \bm already defined.

Yes somehow defining a command in math mode is working, but only for the current math block in LaTeX. So that is a bummer.

Conditionally commenting this import out with r if(!knitr::is_html_output()) ' % ' \newcommand{... would solve this again, but it becomes way more verbose again. And, as you've mentioned, using ````{=html} works, but doesn't get interpreted by RStudio.

Rendering a document for both HTML and PDF with this is ok, but it is just the RStudio IDE preview not able to know that some command are defined for MathJax. I don't think the IDE can do better than that unfortunately.

IDE visual editor does a better job at previewing because it will replace any defined command in the document by its definition in the md source. You can try to see how it works.

[cderv]

For this specific use case, we could have a much simpler feature we could support, again based on what we do in Quarto (https://quarto.org/docs/authoring/markdown-basics.html#equations)

Prototype is this, with a really simple Lua filter

---
title: "My maths"
output: 
  pdf_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("hidden.lua")
  html_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("hidden.lua")
---

```{cat}
#| engine.opts:
#|    file: hidden.lua
#| echo: false
Div = function(div)
  if string.sub(FORMAT, 1, 4) ~= "html" and div.classes:includes("hidden") then
      return {}
  end
end
```

```{=latex}
\newcommand{\e}{\mathcal{E}}
```

::: {.hidden}
$$
\newcommand{\e}{\mathcal{E}}
$$
:::

$$
\e
$$

This probably be worth it to add this feature in R Markdown for all format.

[jassler]

Thank you for pointing me towards Lua filters, I didn't know about them!

With absolutely no experience in Lua, I've tried to come up with a vastly over-engineered solution to keep my \newcommand-s in sync with the html and tex output. However, there are three issues:

1. my solution does not allow empty lines in the .hidden block (not sure why),
2. the commands.tex file has to be created before-hand, and
3. compilation requires two passes when the hidden block changes. (I suppose it comes from the intermediate step of writing the Lua-file and then executing the Lua-file or something)

Not sure if I should feel proud about the solution. It was much simpler without the filtering part.

---
title: "My maths"
header-includes:
  - \usepackage{bm}
output:
  html_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("extract_commands.lua")
  pdf_document:
    pandoc_args: !expr rmarkdown::pandoc_lua_filter_args("extract_commands.lua")
    includes:
      in_header: "commands.tex"
---

```{cat}
#| engine.opts:
#|    file: extract_commands.lua
#| echo: false
WriteCommandsFile = function(text, exclude)
  local file = io.open("commands.tex", "w")
  for line in text:gmatch("[^\r\n]+") do
    local append = true
    for i = 1, #exclude do
      if line:match("\\newcommand{\\" .. exclude[i]) then
        append = false
        break
      end
    end
    if append then
      file:write(line .. "\n")
    end
  end
  file:close(file)
end

Div = function(div)
  if string.sub(FORMAT, 1, 4) ~= "html" and div.classes:includes("hidden") then
      WriteCommandsFile(
        div.content[1].content[1].text,
        {"bm", "otherCommandsNotAvailableInMathjaxButAvailableInLatex"}
      )
      return { }
  end
end
```

::: {.hidden}
$$
\newcommand{\e}{\mathcal{E}}
\newcommand{\bm}[1]{\boldsymbol{#1}}
$$
:::

$\bm{\e}$

[cderv]

compilation requires two passes when the hidden block changes. (I suppose it comes from the intermediate step of writing the Lua-file and then executing the Lua-file or something)

I think it is because you are writing in the file from the Lua filter. So it is too late for Rmarkdown and/or Pandoc to correctly include the tex file as header.

What are you trying to do exactly here ? I see otherCommandsNotAvailableInMathjaxButAvailableInLatex : so if you want to include some commands only for LaTeX you can do that in the YAML directly See in https://github.com/rstudio/rmarkdown/issues/2457#issuecomment-1448209029

header-includes:
  - |
    ```{=latex}
    \newcommand{\e}{\mathcal{E}}
    ```

and if you want to include a file, just pass it to pdf_document(includes = ) - see doc: https://bookdown.org/yihui/rmarkdown/pdf-document.html#includes-1

the commands.tex file has to be created before-hand

Use the text YAML field in this case or use a cat chunk inside your document to write from within the Rmd.

my solution does not allow empty lines in the .hidden block (not sure why),

Not sure. But I don't think you need a Lua filter for what you are trying to do in addition to the previous hidden feature I shared.

Or maybe it is that you want to use the content of the of the hidden block to create some includes to pass in the LaTeX file without having to duplicate those command in a latex preamble beforehand ?

Is that so ?

[jassler]

Or maybe it is that you want to use the content of the of the hidden block to create some includes to pass in the LaTeX file without having to duplicate those command in a latex preamble beforehand ?

Yeah, exactly! In classic programmer fashion, I wanted to avoid copy-pasting the commands. Instead, I wanted to auto-generate a preamble with \newcommand-s based on the contents of the .hidden block. That's why I tried to filter out commands such as \bm, since they are already defined in Latex but have to be defined for Mathjax.

So, for the example above, while the .hidden block contains

::: {.hidden}
$$
\newcommand{\e}{\mathcal{E}}
\newcommand{\bm}[1]{\boldsymbol{#1}}
$$
:::

, the generated commands.tex file only contains

\newcommand{\e}{\mathcal{E}}

(I'm also not sure where the dollar signs went, or why div.content[1].content[1].text didn't include them)

I have about 30 commands defined and frequently have to change small pieces, so I thought having this convoluted setup may be justified.

[cderv]

(I'm also not sure where the dollar signs went, or why div.content[1].content[1].text didn't include them)

I think you need to look at the AST to understand. The dollars won't be there as Pandoc will probably parse as math. Or you went into the math object with div.content[1].content[1] and get only the text not the whole object

I have about 30 commands defined and frequently have to change small pieces, so I thought having this convoluted setup may be justified.

But you would still have to maintain the list of commands to excluse in the Lua filter right ? So write both in the $$ block for mathjax and in the lua filter ?

But yes I can understand why you do that. I think just little tweak is needed for the space to be taken into account.

but the two pass will still be necessary.

Let's recall that all that is needed for you because you want to keep the source preview working in IDE and still produce HTML + PDF

If you don't need preview you could have one file with command for HTML and another with command for PDF and then include each in its format.

[jassler]

But you would still have to maintain the list of commands to excluse in the Lua filter right ?

Yeah, that's true. Though the filter part is not much of a hassle, it only affects one command so far.

And, you're right, it might be worth it in the end to just focus on the html document and work on a pdf-version separately. I'll see, if my current setup holds up for me (and if future updates to rmarkdown offer alternative solutions 😋)