ENH: Update Dependencies to incorporate Downstream Improvements in myst-parser and myst-nb

[mmcky]

trafficstars

This PR was initiated by @cadr and just making a local branch version here for additional updates:

• [x] pin msyt-nb to 0.16
• [x] update sphinx-jupyterbook-latex to support improvements in myst-parser and myst-nb
• [ ] fix deprecation notices (CI checks)
• [ ] fix configuration updates in docs (CI checks)
• [ ] fix unknown mime type warning (mystnb.unknown_mime_type)
• [ ] fix broken links (CHANGELOG.md:306: WARNING: 'myst' reference target not found: [#1094])

Tasks Remaining:

• [ ] Update documentation to use latest configuration names specified by myst-nb and myst-parser
• [ ] Update the documentation to reflect the breaking changes
• [ ] Update the documentation to reflect the new additions/features

#1819

[mmcky]

I have misunderstood the deprecation notices that are issued on the CI so I will need to look at myst-nb docs to understand what the new configuration names and structure is.

[codecov[bot]]

Codecov Report

Base: 91.39% // Head: 91.42% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (8de9aea) compared to base (bb1a1b4). Patch coverage: 85.71% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1842      +/-   ##
==========================================
+ Coverage   91.39%   91.42%   +0.02%     
==========================================
  Files           7        7              
  Lines         686      688       +2     
==========================================
+ Hits          627      629       +2     
  Misses         59       59              

Flag	Coverage Δ
pytests	91.42% <85.71%> (+0.02%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jupyter_book/cli/main.py	95.09% <ø> (-0.04%)	:arrow_down:
jupyter_book/config.py	94.32% <83.33%> (+0.11%)	:arrow_up:
jupyter_book/__init__.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[AakashGfude]

Have suppressed the warning of 'sphinx_proof.domain::prf' has not implemented a 'resolve_any_xref' method [myst.domains] for now. We need to implement it in sphinx proof.

[choldgraf]

Just want to voice my enthusiasm for this update - we really need to bring in these latest changes to the Jupyter Book docs especially since there have been some significant changes lately, so please let me know what I can do to help you move this forward @mmcky and @AakashGfude

[AakashGfude]

Do we want to give a priority to mime_types:application/javascript and text/html? I will have a look at the pdf outputs if I do define in _config. Or we can just ignore the warning, like the present state of this PR.

But, I will check what the output is like, with or without.

EDIT: pdf anyways doesn't display these mime types duh.

[AakashGfude]

@choldgraf Not sure what tests should be added here to make the coverage/patch run, as this PR just does changes in docs, variable names and config

[AakashGfude]

Apart from that the build is running. @mmcky are the docs updated here, or do we need to update things?

[choldgraf]

I'd focus on getting the tests to pass and making the docs updated. The codecov check isn't an issue if we know that it doesn't represent a significant regression in coverage IMO

[mmcky]

Apart from that the build is running. @mmcky are the docs updated here, or do we need to update things?

Hey @AakashGfude we will need to do a full pass of the docs as there are a number of changes from underlying updates.

[AakashGfude]

@mmcky @choldgraf I have looked through the docs, and have done small changes which I thought were important. Let me know how it looks.

[mmcky]

@AakashGfude with a number of the configuration changes I think we need to write a short section around updates needed when moving from jupyter-book<0.14 to jupyter-book>=0.14 with some common gotcha's and how to fix them.

For example:

execution_allow_errors -> nb_execution_allow_errors

with an example of the warning that is thrown.

We should also add some discussion around:

• [x] behaviour of configuration passthrough myst-parser and myst-nb

I think this is likely to be the biggest issue for users when changing over.

[AakashGfude]

Hi @mmcky , did you have something like this in mind https://github.com/executablebooks/jupyter-book/pull/1842/commits/c89552f00feac3aa54dcb57d9869ad6a8a5a756f , when highlighting migration changes? Just wanted to see if I am on the same page, if yes, then i will go ahead with highlighting changes(mostly which were necessary when migrating the docs)

[alcrene]

Maybe this is already planned as part of the update to the docs, but might I suggest adding to the “Migrating to jupyter-book>=0.14” page some comments regarding the changes to the glue framework ? For my documents at least, the fact that we can no longer glue across documents (see MyST-NB #431) will require some workflow changes.

If you can include semi-official workarounds for glueing (esp. figures and numerical values), that would be even better, but I understand if that is still WIP.

[mmcky]

but might I suggest adding to the “Migrating to jupyter-book>=0.14” page some comments regarding the changes to the glue framework ?

thanks @alcrene. This is a great suggestion.

[AakashGfude]

Maybe this is already planned as part of the update to the docs, but might I suggest adding to the “Migrating to jupyter-book>=0.14” page some comments regarding the changes to the glue framework ?

Hi @alcrene , we have already added the changes to glueing across documents in this PR here: https://github.com/executablebooks/jupyter-book/pull/1842/files#diff-b29b3ce9aba14e42df512e99c8742d7080f1b82e2b4f2959f6fbf259bfa52203R118 . This works in jupyter-book docs though, like we have done in this PR here: https://github.com/executablebooks/jupyter-book/pull/1842/files#diff-5bac10dd315f669536460e8240e2cabdb1c006b5e878ad0d1f83aaeddb094f0bR181 . You can check the preview of the website to verify. The comments in that thread you mentioned are from a while back, so probably the issue of glueing figures has been taken care of by now?

[mmcky]

@AakashGfude something for sphinx-jupyterbook-latex:

WARNING: Fontawesome icons not included in LaTeX output, consider 'sd_fontawesome_latex=True' [design.fa-build]

• [x] ~~update sphinx-jupyterbook-latex to enable the option in sphinx-design https://sphinx-design.readthedocs.io/en/latest/badges_buttons.html?highlight=fontawesome#fontawesome-icons~~ enable this option via docs/config

[mmcky]

• [x] docutils-0.17.1 is still being installed into the test environment so there must be another package with docutils ... <0.18 (Update: Ah it's actually sphinx -- Once we get sphinx5 compatibility this will work)

[choldgraf]

Hey all - just a note that I updated the title of this PR to reflect that it is primarily upgrading a bunch of dependencies rather than releasing a new version of Jupyter Book. I think once this PR is merged (and potentially after a new book theme update as well), we should do some release candidates since these are a lot of changes.

[alcrene]

Hi @alcrene , we have already added the changes to glueing across documents in this PR here: https://github.com/executablebooks/jupyter-book/pull/1842/files#diff-b29b3ce9aba14e42df512e99c8742d7080f1b82e2b4f2959f6fbf259bfa52203R118 . This works in jupyter-book docs though, like we have done in this PR here: https://github.com/executablebooks/jupyter-book/pull/1842/files#diff-5bac10dd315f669536460e8240e2cabdb1c006b5e878ad0d1f83aaeddb094f0bR181 . You can check the preview of the website to verify. The comments in that thread you mentioned are from a while back, so probably the issue of glueing figures has been taken care of by now?

(To be clear: everything I say below is based on the checkout of this PR I made last Thursday, commit #85f133c76f)

Somehow I missed the comments regarding glue, even though I made sure to compile the new docs before commenting this issue. Sorry for that. Nevertheless, when I try the form suggested in the migration guide,

:::{glue} sorted_means_fig
:doc: executable/output-insert.md
:::

I get

<Figure size 640x480 with 1 Axes>

in the generated page. From my limited understanding, the new glue system exports data in a Sphinx node which basically can only contain simple data like plain text and numbers, so it makes sense that rich data like a plot would not survive. (So my interpretation is that what we see is the text MIME type of the figure object.) Glueing the same figure within the notebook where it is created works as expected, displaying the figure graphically. Note also that with regards to cross-document glueing, the current docs for MyST-NB only document the {glue:text} directive.

Critically, the suggested form with :doc: doesn't use the {glue:figure} directive, and so doesn't allow specifying a caption. (One gets the error Directive 'glue': No content permitted.) So in terms of functionality it can do less than the previous form, and is insufficient for almost any scholarly writing. As far as I can tell, figure attributes, like :width: and :name: for cross-references, are also no longer supported. This makes sense, since the generic {glue} would not know about these attributes.

What does work, and work quite well, is to simply save the figures to some common /figures directory, and then insert them using a normal :::{figure} directive. I honestly struggle to think of any benefit glueing figures could provide over this approach, so while I've had to reformat my documents a bit, I'm quite happy to do it this way. It's certainly more robust and easier to debug incorrect references.

At this point I've probably logged 3+ hours of experimentation to figure out the subtleties of the new glue. (This is not a complaint ! I'm fully aware that this is an unmerged PR :stuck_out_tongue: ) The main things I've taken away are the following:

• Glueing within documents works as before.
• Glueing across documents works for plain text and numbers. (In contrast to what I said above, numbers work fine.) However:
  • One needs to prepend the file path to the key (this is nicely indicated in the message output and the MyST-NB docs): {glue:}`path/to/file.ext::name`
  • The file extension is required. The following does not work: {glue:}`path/to/file::name`
  • When using {glue:text}, a trailing colon is required. (This might be a bug ?)
    The following does not work: {glue:text}`path/to/file.ext::name
    The following forms do work: {glue:text}`path/to/file.ext::name:, or {glue:text}`path/to/file.ext::name:.2f
• Glueing across documents might work for the {glue:math} role (I haven't tested).
• Glueing across documents does not work for the {glue:figure} directive. The plain {glue} directive combined with :doc: works (in that no error is generated), but only inserts plain text. Saving figure images to an export directory is a better solution.

Anyway, I'm really sorry for harping about this one feature, but the good news is that it's the only one thing that tripped me when I upgraded my JB to the version in this PR. Every other change was seamless, requiring little to no effort. Hopefully this migration experience is helpful to you.

[mmcky]

@alcrene your feedback on this is super helpful. Thank you ❤️

[Zethson]

I hope that this PR also includes the fix for https://github.com/spatialaudio/nbsphinx/issues/687

I consider this major. See https://github.com/executablebooks/jupyter-book/issues/1896 for context

[AakashGfude]

The sphinx-book-theme and pydata-sphinx-theme version updates are working fine now from what i checked in the preview. Let me know @choldgraf , @mmcky if you find anything breaking.

[mmcky]

thanks @AakashGfude do we need to organise releases for the underlying packages?

[AakashGfude]

@mmcky there is a pre-release for sphinx-book-theme which can work with this PR. For a proper release we need to finish this issue first: https://github.com/executablebooks/sphinx-book-theme/issues/670. I reckon I should put more attention to that.

[mmcky]

Note: these issues seem to be related to sphinx-book-theme==1.0+ so I have reverted to using sphinx-book-theme==0.4.0rc1

• [ ] @AakashGfude on Monday would you be able to review the sphinx-multitoc-numbering changes to fixtures and confirm they are valid and accurate.

  I think the main difference is <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"> goes to <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/> a closing slash before the last >

• [x] fix pdflatex errors -- interacting with sphinx-book-theme

  @AakashGfude @choldgraf has there been a change in sphinx-book-theme that looks for a theme_options object?

  Extension error (sphinx_book_theme):
  E         Handler <function update_mode_thebe_config at 0x108274f70> for event 'builder-inited' threw an exception (exception:     'LaTeXBuilder' object has no attribute 'theme_options')
  

  It is odd as sphinx-book-theme shouldn't influence the pdflatex build pathway.

  OK I think this is failing due to the latest rc for sphinx-book-theme https://github.com/executablebooks/sphinx-book-theme/pull/683/files#r1115221983

[mmcky]

thanks @alcrene for your feedback on glue it was super helpful and I have gone through the docs to update the issues you have noticed with glue.

@chrisjsewell I have made some comments in this commit (https://github.com/executablebooks/jupyter-book/pull/1842/commits/7316e323ef533153967898ebdd76082e4c5caa18) that document some issues with the glue directive when used between documents. I will raise some issues on myst-nb and the outputs can be seen here

[fperez]

Hi, I'm curious as to whether folks think this might land soon? I'm teaching a course where we make heavy use of jupyterbook, but also plan on using myst and widgets. Unfortunately 0.13.2 won't install with current versions of widgets, and finding a working combination is proving to be quite tricky.

I greatly appreciate all the work that's going into the project, so this is just me asking so I can better plan how to proceed for the rest of the semester. Many thanks to the whole team!!

See also #1898 by @consideRatio.

[chrisjsewell]

Heya @fperez, I think @mmcky basically had this good to go 😄 ?

[fperez]

Hi @chrisjsewell, thanks! That would be awesome. I've concluded it's not possible to get a working combination of jbook, myst and widgets that showcases the latest interactive features in myst 1.1. I was able to get all the packages to install after much trial and error, but ultimately it requires downgrading to old enough versions of the widgets machinery that none of the new myst features work :(

So we're currently stuck in a situation where we can either use jbook or cutting-edge myst, but not both. It would be phenomenal if this could go out soon - I can update our production hub very quickly once there's a package out (preferably conda-forge, though I can manage with pypi if need be).

Again - no pressure, I understand people are all doing their best and I'm very grateful.

[mmcky]

@fperez just planning a final round of the docs updates tomorrow (as there are a few breaking changes in this release) and then plan to get this merged. I hope to get this out early this week.