Tag to skip cell execution

[wcbeard]

Is your feature request related to a problem? Please describe.

There are some cells I'd like run during development, but not during the book build step. The example I'm working with now are magic calls.

Describe the solution you'd like I would like a tag that I can add to a cell, not to only hide the cell input/output (as here--it appears that even remove-cell still executes the contents), but a tag that ensure the entire code cell isn't run during build.

Describe alternatives you've considered I've tried commenting the cells out when I'm doing making interactive changes, and uncommenting them if I have to do some interactive work.

Additional context n/a

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
Welcome to the EBP community! :tada:

[najuzilu]

I've used :class: no-execute for code-blocks in rST files (see example here). I wonder if that works as a tag in a notebook.

[chrisjsewell]

Thanks @wcbeard,

@najuzilu no this will not work I'm afraid. Currently, the full notebook is parsed for execution to nbclient. Skipping cells via tags then would either require an implementation in that package (similar to raises-exception), removing the cell before execution (not ideal), or executing the notebook in an alternate manner.

The other option would be to add code to the cell(s), like cell magics or "if" statements, to control whether the code is called.

Some relevant links:

• https://github.com/ipython/ipython/issues/11582
• https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/freeze

[choldgraf]

I also opened up an issue here to see if anybody else has used a pattern like this: https://discourse.jupyter.org/t/is-there-a-cell-tag-convention-to-skip-execution/5445

ping @jni as well because he was just mentioning this to me the other day!

[chrisjsewell]

I also opened up an issue here to see if anybody else has used a pattern like this: discourse.jupyter.org/t/is-there-a-cell-tag-convention-to-skip-execution/5445

But still the issue would be there is no (current) way to actualize that tag in the execution process

[choldgraf]

yeah - I agree that's something we'll need to figure out, just wanted to see if it's a pattern anyone else has already worked-around before

[jni]

Wow, timing. Yup, we'd love this at scikit-image/elegant-scipy. The motivation is to have some "exercise" cells for readers, such as:

image_rgb = data.astronaut()
r = ...  # Grab the red channel
plt.imshow(r)

What does "actualize" mean in this context? Why is it an issue?

For me, the raises-exception+remove-output combination workaround suggested by @choldgraf works, because these things don't take long to execute, but it feels like a hack, and execute-off would feel much more natural.

[choldgraf]

I wonder if, in lieu of actually updating something like nbclient, we could support a tag like execute-off that would basically behave like it was raises-exception + remove-output ?

[jni]

Well, then the tag would be lying, wouldn't it? =) I'm picturing:

-----
tag: execute-off
-----
!rm -rf .

=P

[chrisjsewell]

What does "actualize" mean in this context? Why is it an issue?

Currently, the code for executing a notebook is:

from nbclient import execute
ntbk = execute(ntbk)

There's no external way to include logic for skipping execution of specific cells.

IMO Ideally there would be a hook near the start of nbclient's async_execute_cell method, to inject a function which decides if the cell should be executed, e.g.

def pre_execute_hook(cell):
  """Modify the cell in-place before execution.

  :returns: True if the cell should be executed, False otherwise.
  """
  return 'execute-off' not in cell['metadata'].get('tags', [])

ntbk = execute(ntbk, pre_execute_hook=pre_execute_hook)

Otherwise, we have to start overriding parts of nbclient, which is not ideal.

Edit: This is almost what I had in mind: https://github.com/jupyter/nbclient/pull/79, so might suggest this there

[choldgraf]

yeah - thus far nbclient has generally been resistant to changing a lot of its functionality, and has opted instead to be a very vanilla execution engine. I think the "fancier" or more customizable stuff is being put in papermill.

Though something like pre- and post-execution hooks seem like they could be in-scope for nbclient. That said, when I've brought stuff like that up before the suggestion was to do it in papermill

[chrisjsewell]

See https://github.com/jupyter/nbclient/pull/79#issuecomment-667543868

[mmcky]

The way we previously addressed this in sphinxcontrib-jupyter was to remove the code with tag no-execute prior to notebook execution. (as @chrisjsewell says we can't adjust the notebook once passed to nbclient -- unless implement code-cell skipping upstream).

It was easier for sphindcontrib-jupyter though as we generated different notebook states for the different output targets (html generation, code execution tests for execution testing).

We will will need some tag based control for quantecon projects for things like:

• [ ] no-execute
• [ ] test (for adding code based tests of notebook content but don't want it to be displayed in the output)
• [ ] hide-output (already done for html)

[choldgraf]

It seems like a workable step could be to convert the cell type to "raw", pass to nbclient, then convert back when it is returned

[chrisjsewell]

It seems like a workable step could be to convert the cell type to "raw", pass to nbclient, then convert back when it is returned

Well let's wait while I work with the guys on https://github.com/jupyter/nbclient/pull/79, before we start doing anything hacky 😉

[jni]

It seems like a workable step could be to convert the cell type to "raw", pass to nbclient, then convert back when it is returned

Well let's wait while I work with the guys on jupyter/nbclient#79, before we start doing anything hacky 😉

fwiw I find this particular hack quite elegant. =)

[wcbeard]

Is there a way to detect when a cell is being run from my regular Jupyterlab kernel, versus when it's being run from jupyter-book build? I think a temporary shortcut for me could be to wrap it in a conditional based on this context. It looks like __name__ == '__main__' in both cases, but are there any other differences in the environment that I could use to figure out the context?

[choldgraf]

hmmm that's a great question, I am not sure. I think that'd be a question for nbclient (https://github.com/jupyter/nbclient) since that's what we're using to run the notebook.

[fm75]

An interesting variation on this. I wanted to execute a code cell that is known to fail but continue executing the notebook regardless. I wanted the book reader to be able to see how something fails in the output. My work-around was to try: the code with an error and print(sys.exc_info())

[choldgraf]

@fm75 see https://jupyterbook.org/content/execute.html#dealing-with-code-that-raises-errors

[fm75]

The {code-cell} annotation works in the example, but the source is a markdown file, not a notebook. (The source markdown file also has a 4 back-ticked md surrounding the code cell syntax)

I tried it both ways in a notebook code cell. Each results in a SyntaxError on execution while running the notebook as well as when it is executed in the build.

[chrisjsewell]

I'm guessing that maybe you haven't quite understood the Markdown-to-Notebook mapping yet 😬

You don't use the {code-cell} annotation in a notebook, that gets converted to a notebook cell before execution. The one with the 4 back-ticks surrounding it is not the actual cell, is just to show the literal syntax. If you download the notebook, using the button at the top of the page, you will get the notebook version.

and see the cell:

{
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "remove-stderr"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is some stdout\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "this is some stderr\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(\"this is some stdout\")\n",
    "print(\"this is some stderr\", file=sys.stderr)"
   ]
  }

that's what the {code-cell} gets converted to (and the outputs added during execution).

Or you can convert between them using jupytext: jupytext --from myst --to ipynb myfile.md

[fm75]

Perhaps. I have been building both markdown files and actual notebooks in binder, pushing them back to a repo, then building/publishing.

If I understand, the use of sphinx directives should be avoided in notebooks? Or is there some clever markdown syntax that I can use at the start of a notebook to make this work?

Are you suggesting that I should only be working in msyt styled markdown and converting that to .ipynb? That seems pretty indirect. I like working in notebooks. The idea of converting them to markdown, followed by enhancement seems like something that I would only want to do as a pretty special case.

Thanks for your help with this. I have only been working on this for less than a week, so trying to get up to speed. I have solutions now to two other issues that I encountered, both related to being explicitly tied to GitHub. Both are in sphinx-book-theme. I hope to be allowed to create PRs for them soon.

[phaustin]

That's basically our workflow. We make jupytext the notebook ContentsManger and consider the myst-markdown file to be cannonical. Jupytext automatically updates paired py:percent and ipynb files on every change. We run black and flake8 on the py:percent file and use it for continuous integration testing. We treat the ipynb file as read-only output-- we don't check json files into git. There's no need to have rendered ipynb files on github because we've got the rendered jupyterbook html on github.io using gh-pages.

[chrisjsewell]

If I understand, the use of sphinx directives should be avoided in notebooks?

No they can, Markdown is parsed exactly the same, by jupyter-book, in notebooks cells as it is in the Markdown files, it should make no difference whether you write in Markdown files or Notebooks 😄

[fm75]

Thanks. I see, now, how to start with a notebook, use jupytext in the notebook to generate the markdown. It seems like a reasonable "round-trip" for development. :)

[akhmerov]

A related issue with a different user story jupyter/jupyter-sphinx#140.

It seems like a workable step could be to convert the cell type to "raw", pass to nbclient, then convert back when it is returned

Converting cells to raw and back would lose the outputs if those are present, wouldn't it? An alternative low-level approach would be to take those cells out and put them back in place after execution.

[mmcky]

I think a tag for this would be great. FWIW - in sphinxcontrib-tomyst I have been adding support for targeting code-cell directives for execution and using the sphinx code-block directives for static code that is not to be executed. The code-block get's added to the markdown cell when converting to notebook via myst-nb

We used :class: no-execute in our rst for quantecon projects to distinguish between blocks that do and don't execute.

[chrisjsewell]

FYI on this, https://github.com/jupyter/nbclient/pull/151 is now merged, so once a new version of nblient is out, you will be able to use the skip-execution tag 😄

[chrisjsewell]

This is now available from nbcient>=0.5.5 (https://github.com/jupyter/nbclient/compare/0.5.4...0.5.5) I believe this is installable with latest jupyter-book (it should be only jupyter-cache pinning nbclient: https://github.com/executablebooks/jupyter-cache/blob/7917c680f97af055ac60fca345822d092f1d3d6e/setup.cfg#L32) So closing this issue then should just be a matter of adding some documentation on it (around here: https://jupyterbook.org/content/execute.html#dealing-with-code-that-raises-errors, and also mirrored here: https://myst-nb.readthedocs.io/en/latest/use/execute.html#raise-errors-in-code-cells)