nbsphinx strips backslash from color keyword in array math environment

[phaustin]

trafficstars

We've encountered an issue where nbconvert succeeds and nbsphinx fails in a array equation environment. Specifically, this notebook markdown:

https://github.com/phaustin/nbsphinx_issue/blob/master/Lec5_Euler_debug.py#L54-L63

Is successfully translated in to \color calls by jupyter nbconvert --to html

https://github.com/phaustin/nbsphinx_issue/blob/master/Lec5_Euler_debug_nbconvert.html#L11804

but nbsphinx removes the backslash:

https://github.com/phaustin/nbsphinx_issue/blob/master/Lec5_Euler_debug_nbsphinx.html#L369

[mgeier]

Thanks for the report!

How did your create the output?

Can you please provide the Sphinx configuration that you were using?

[phaustin]

Added conf.py and build.sh to:

https://github.com/phaustin/nbsphinx_issue/commit/1ca7c2cca4368fba5750d44bf2ad57d76fff9e65

[mgeier]

Thanks for the update!

So this error is the result of a work-around I recently created (#257) to work around problems that came into being after some changes in pandoc.

As a quick fix, you can remove all indentation from your LaTeX equation, then it should work:

\begin{equation}
\frac{\color{blue}{c_{\text{TMF}}}(t + \Delta t)-c_{\text{TMF}}(t)}{\Delta t} =  \frac{Q_{\text{pit}}  c_{\text{pit}} + Q_{\text{mill}}  
c_{\text{mill}} + A_{\text{bottom}} k \left( c_{\text{pore}} -   {\color{red}{c_{\text{TMF}}}}  \right)  
-Q_{\text{dis}} \color{red}{c_{\text{TMF}}}}{V_0}
\end{equation}

I don't know yet what's the proper fix for this, I'll think about it. If you have an idea, please tell me!

There is a separate, unrelated problem in your example: You are using a space after an opening $. You should change $ t + \Delta t$ to $t + \Delta t$. That's a known (and reasonable) limitation of pandoc, see https://nbsphinx.readthedocs.io/en/0.4.2/markdown-cells.html#Equations.

BTW, it's nice that you are using nbsphinx_custom_formats. I wasn't sure if anybody would be interested in that and I've never heard any feedback ...

[phaustin]

yes, that fixed it -- many thanks.

https://github.com/phaustin/nbsphinx_issue/commit/410c06be14f16a2ecc5fb284ae634715380d90b7#diff-75cd95cf232b668a98307e4d7c6242c9

Re -- custom_formats and jupytext --I think the round-trippable python format is very big deal for both research and teaching (and that diff above is a concrete example). We'll be adopting nbsphinx/jupytext for all of the notebook based material in our department.

[mgeier]

I'm trying a different approach for replacing the JSON-based notebook format: https://github.com/mgeier/jupyter-format/

This is how your eosc213_students project would look like when using my new format: https://github.com/mgeier/eosc213_students

What do you think about it?

[phaustin]

It's definitely readable, but are there other benefits that would make it a substitute for jupytext py:percent? For us, making the primary repo format executable python has the major advantage that we can seamlessly move the students from notebooks to editing scripts/modules using vscode or pycharm. Our students are also a mixture of R and python users, so being able to let them use Rmd if they want for their projects a plus. So if .jupyter was compelling then our workflow would probably be .jupyter -> py:percent/Rmd -> ipynb

[mgeier]

are there other benefits that would make it a substitute for jupytext py:percent?

Arguably, my format is more readable because you don't have # everywhere.

But I'm not really trying to make a substitute for py:percent (which has valid use cases), I'm trying to make a substitute for the original JSON-based .ipynb format.

AFAIK py:percent cannot store output cells, right? I'm looking for a full .ipynb replacement, including output cells.

For us, making the primary repo format executable python has the major advantage that we can seamlessly move the students from notebooks to editing scripts/modules using vscode or pycharm.

I agree that it's very useful to convert notebooks to Python scripts, but I don't see the necessity for enabling a round-trip back from Python scripts to notebooks.

let them use Rmd if they want for their projects a plus.

Sure, if people want to use Rmd they should be able to, and there is nothing speaking against it. jupytext is a great tool to enable that, and with nbsphinx_custom_formats this can also be used in nbsphinx.

But Rmd is IMHO not a reasonable full replacement for .ipynb files.

My idea is to fully replace the JSON-based .ipynb format in all official Jupyter tools at some point (in the far future).

[phaustin]

I see -- for us there are two main features we get out of making python the archival storage format for notebooks:

1. clean diffs for version control -- we run black, flake8 and reorder_python_imports with a pre-commit hook. Having the output stripped, and whitespace etc. standardardized really helps with git history.

2. using real editors for code refactoring, global search and replace. The vscode codelens that recognizes the py:percent format is pretty nice.

We keep the ipynb files with output in a separate directory and typically don't commit them to a repo. The exception is for distribution to students, who don't have jupytext in their environment. If we do commit an ipynb file, we break the pairing with the python file.

[mgeier]

OK, so you are really treating it like Python code.

It would be interesting though, to have notebook-aware versions of the tools you mentioned. I guess this would work well for your point (1) because the tools could work on single cells, but it would be harder for point (2).

But for point (2) you wouldn't really have to use py:percent in version control. You could as well locally use something like this (starting from the traditional notebook format):

1. remove outputs
2. convert to py:percent (which I guess removes outputs anyway)
3. do your code manipulations
4. convert back to notebook format

If this could be somehow (partially) automated, you wouldn't need to save your files in py:percent format.

[phaustin]

for sure -- personally a thing that might move me to add a new format would be large-project, publication-quality features -- especially table/equation/section cross-referencing across notebooks and conditional compilation flags for book formats. Once you get to the scale of http://people.duke.edu/~ccc14/sta-663-2018/index.html the limitations of notebooks really start to matter.