Hidden cells with hide_input argument

[J535D165]

Hello,

Jupyter notebooks do have the ability to hide a cell with "hide_input": true metadata. I think, this should be handled by nbsphinx in the same way as "nbsphinx": "hidden".

My suggestion:

Extend

{%- if cell.metadata.nbsphinx != 'hidden' %}

with

{%- if cell.metadata.nbsphinx != 'hidden' or cell.metadata.hide_input is not True %}

Bye, Jonathan

[mgeier]

This is a duplicate of #15, see also #17.

This could be implemented in nbsphinx, but it is not the same as "nbsphinx": "hidden"! Former is supposed to hide only the input, latter hides both the input and all outputs (and it works on all cell types, not only code cells).

Apparently, the hide_input metadata is not (yet?) part of the official format: https://github.com/jupyter/notebook/issues/534. Does this work in nbconvert? There may be some info at https://github.com/jupyter/nbconvert/issues/163.

It looks like this is supported with a Notebook extension: https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/hide_input.

Should there be a possibility to toggle the visibility of the input cell in HTML output or should it be completely inaccessible? What should be the behavior for LaTeX output?

I'm reluctant to implement this before it's implemented in the Notebook itself and in nbconvert, but if somebody provides a PR, I'll consider it.

[mpacer]

Btw, per comments at the Fall 2016 dev meeting we added a jupyter: prefix to the metadata tags that we were going to give meaning to in order to avoid violating custom user namespaces.

Is there an equivalent of the distinction between hide and kill for nbsphinx; for html output hide just makes it toggleable and defaults to toggled "off" and "kill" actually removes it from the DOM? In PDF only kill had any meaning (since making an object in a pdf that could toggle its visibility would break the paging model of a pdf).

Also, tags are supposed to be included in a list of strings in under the metadata.tags key i.e.,

"metadata": {
    "tags": [
        "jupyter:hide_cell"
        ]
    }

so "nbsphinx": "hidden" isn't technically a tag in the sense jupyter has been using it but a metadata attribute. Specifically, tags aren't associated with a second value denoting state, they are either present or not.

[mgeier]

@michaelpacer Can you please point me to the documentation of the "tag" feature and the available official tags?

I'm not sure if a tag with a binary value (on/off) is the most meaningful way to describe the hiding mechanism, since this involves multiple mutually exclusive states.

Is there an equivalent of the distinction between hide and kill for nbsphinx [...]?

Not yet. Currently, "nbsphinx": "hidden" is equivalent to kill. First, all cells are executed and then the "hidden" cells are completely removed.

One reason for not having it toggleable in HTML was that nobody made a PR with the HTML/CSS code. Another reason is that it is meaningless in LaTeX output. Finally, a third reason is that hiding code isn't a good idea in the first place. We should use the available methods for abstraction/encapsulation instead. In Python this would mean using modules and import statements or using the %run magic.

Remember: "Explicit is better than implicit".

If the code needs to be hidden, I think Jupyter Dashboards might be a better solution, right? Those do not use the notebook format, right?

so "nbsphinx": "hidden" isn't technically a tag

Sure. When I introduced it, the concept of tags didn't exist yet. It was meant as a temporary solution until there was an official mechanism for hiding etc.

And as I alluded to above, binary "tags" or "flags" might not be the best tool anyway. Cells might have metadata about "hide", "hide input", "hide output", which are mutually exclusive. On top of that, some of those might be "toggleable" with "default on" or "default off".

Anyway, whatever gets implemented in the Notebook and in nbconvert should probably eventually also be implemented in nbsphinx ...

[mgeier]

per comments at the Fall 2016 dev meeting we added a jupyter: prefix to the metadata tags

I don't think this was a good idea. Using tags for hiding stuff is already very awkward, having to use the clumsy prefix (for built-in core functions) makes the user experience even worse.

Are those things already fixed or is there still a chance for a saner and more user-friendly solution? Is there still ongoing discussion somewhere?

[mpacer]

Sorry for not responding sooner, I only just saw it today. As you can see, I have plenty of answers :).

@michaelpacer Can you please point me to the documentation of the "tag" feature and the available official tags?

http://nbformat.readthedocs.io/en/latest/format_description.html#cell-metadata

There were some metadata values that were being assigned automatically for some time, but I do not believe there are any official tags more official than what I've been doing. @minrk is there anywhere that lists that?

per comments at the Fall 2016 dev meeting we added a jupyter: prefix to the metadata tags I don't think this was a good idea. Using tags for hiding stuff is already very awkward, having to use the clumsy prefix (for built-in core functions) makes the user experience even worse.

Are those things already fixed or is there still a chance for a saner and more user-friendly solution? Is there still ongoing discussion somewhere?

@mgeier This is probably the "most" active issue right now…but I haven't gotten many responses: https://github.com/jupyter/nbconvert/issues/445

Here's where the work-to-date lives: https://github.com/michaelpacer/hiding_tags_nbconvert but no one's talked to me on that at all.

TBH, I've been waiting on hearing (on the nbconvert issue) how other people were imagining this being implemented beyond what I've done & how they'd like it to be implemented for Jupyter.

I agree that the jupyter: prefix is somewhat awkward and that was why I hadn't originally included it (or any prefix). However, @jasongrout pointed out that we shouldn't be intruding into user namespace (which technically the tags have been for a while, since (as I said above) there aren't really any "official" tags). I see the point to that, and it seems to trade-off against the "awkwardness" of the prefix. One way to compromise, would be to change it to just a "j:" so, j:hide_cell.

However, I agree this should be a broader conversation. @fperez, is there a better location to have this discussion than on the nbconvert issue? It seems like this would be a bigger issue than just nbconvert, but I'm not sure where that discussion should happen?

I'm not sure if a tag with a binary value (on/off) is the most meaningful way to describe the hiding mechanism, since this involves multiple mutually exclusive states.

Yes and no.

First, tags like this aren't only binary, they're purely additive. In contrast "nbsphinx-hidden": True and "nbsphinx-hidden": False would be binary substitutive features.

Additive features work from the assumption of a null space, and so we may be able to use heuristics about how the "null" case works which allows us to avoid making explicit decisions about the full $2^n$ space of potential tag values.

For example, this allows applying a fairly straightforward hierarchy simply by instituting a sequence of if/elseif/…/else statements, see https://github.com/michaelpacer/hiding_tags_nbconvert/blob/master/toggle_hidden.tpl#L4-L21 in which jupyter:hide_cell ≻ jupyter:hide_output ≻ jupyter:hide_input.

Additionally, additive features are good data structures for a tagging like interface for applying metadata to cells without needing to jump into the metadata editing view. This has a number of UI consequences, some of which include an easy "click"-able interface and tag suggestions. To illustrate, designing a tag suggestion system based on substitutive features would require deciding whether to make suggestions on the basis of the number of times a tag is invoked (regardless of it's value) in addition to deciding whether to suggest a value as well.

Cells might have metadata about "hide", "hide input", "hide output", which are mutually exclusive.

NB: Technically there could be a case in which you separately wish to toggle the output and input which would make the jupyter:hide_output ≻ jupyter:hide_input wrong (see issue https://github.com/michaelpacer/hiding_tags_nbconvert/issues/1). I agree that jupyter:hide_cell and (jupyter:hide_input or jupyter:hide_output) are more or less mutually exclusive, but they are "mutually exclusive" where one of the features has a dominance relation over the other two. Note this makes it more of a poset than a fully ordered set and means the beautiful reduction in the dimensionality of the decisions needs to go by the wayside to support stuff like this. Alas, such is life.

On top of that, some of those might be "toggleable" with "default on" or "default off".

That's one of the beauties of additive features! You don't toggle their state, they all default to "not present", because that's the null state(i.e., you never really have a feature that is "off"). From a data structure point of view this is doubly simpler because if you have only substitutive features you will have to make sure that they are always defined, with a default value, clogging up the metadata. Note, that if you had a substitutive feature that you did not include, you actually have now created a tiered feature space, one in which there are binary additive features (present/not present) and one that assigns the values to those features.

In this conceptual scheme: the notion of a tag that defaults to "on" would be a little odd. Such a feature (which should be selectively turned off by manual decisions) is effectively just a feature that is integrated into the notebook and nbconvert machinery directly. Then there would be a tag to turn "off" this behaviour. In fact, you could see the jupyter:hide_cell as being exactly such a feature! The default behaviour is to show the cell ("on") and when this is present by default it is not shown, but hidden and able to be shown with a toggle.

There are plenty of examples that are harder to fit within this scheme, but those can be dealt with using full metadata attributes rather than tags. This isn't reducing expressivity of the entire system, it's just tailoring the expressive capacity of this one data structure to be useful in the particular cases in which we want to use it.

---

One reason for not having it toggleable in HTML was that nobody made a PR with the HTML/CSS code.

Fair enough!

Another reason is that it is meaningless in LaTeX output.

Yes, that's why in my repo I only have a kill_hidden.tplx not a toggle_hidden.tplx.

Finally, a third reason is that hiding code isn't a good idea in the first place. We should use the available methods for abstraction/encapsulation instead. In Python this would mean using modules and import statements or using the %run magic.

I couldn't disagree more, but not because I think it is necessarily a good idea in the first place, but rather that I cannot evaluate whether it is a good idea for all of the cases in which people would want to use it. There are tonnes of people who have been requesting this feature for an extremely long time. I'm not one to suggest that every one of them was making the right call, but I'm also not one to suggest that everyone one of them was making the wrong call. I'm confident that at least one person could write better documentation for their software using toggleable cells/inputs/outputs than they could without.

Even so, I can think of plenty of reasons in pedagogical contexts that have nothing to do with software documentation where encapsulation and abstraction may not make sense as conceptual primitives around which to organise material. In those cases, having this in nbconvert would allow them to do things that they would not otherwise be able to do. @jhamrick, you may be able to provide more concrete examples given your experiences with nbgrader, which I know didn't rely on tags per se but did rely on other kinds of metadata that made things selectively visible. Would there have been any use for toggling?

[mgeier]

Thanks for the detailed response, that's really interesting stuff!

Regarding http://nbformat.readthedocs.io/en/latest/format_description.html#cell-metadata:

What about the collapsed value? You were talking about hide_input and hide_output ... wouldn't hide_output be the same as collapsed? Is collapsed deprecated?

Here's where the work-to-date lives: https://github.com/michaelpacer/hiding_tags_nbconvert but no one's talked to me on that at all.

I think (part of) the problem is that you are approaching this from the nbconvert side, but I think this should be first discussed in the context of the Jupyter Notebook. If certain metadata makes sense or not, depends heavily on the way how and if it is bound to UI elements in the Jupyter Notebook. Either way, it will affect several sub-projects. Could this be a Jupyter Enhancement Proposal https://github.com/jupyter/enhancement-proposals?

One way to compromise, would be to change it to just a "j:" so, j:hide_cell.

This would be even worse than jupyter:hide_cell. It's still awkward, but as an additional disadvantage it's also less explicit. I think it would be best to leave the tags to the users and store the built-in Jupyter metadata in a separate section. And if "additive tags" are really the best data structure, there could be different sets of tags, right?

Additionally, additive features are good data structures for a tagging like interface for applying metadata to cells without needing to jump into the metadata editing view. This has a number of UI consequences, some of which include an easy "click"-able interface and tag suggestions.

Yes, that sounds reasonable for a tagging system. But does hiding/unhiding really fit into this? Shouldn't there be special UI elements to hide/unhide cells? I've already commented on this there: https://github.com/jupyter/notebook/issues/1812#issuecomment-269015278. Would you be able to edit the tags of a hidden cell? If the tags are still there, it's not really hidden, is it?

if you have only substitutive features you will have to make sure that they are always defined, with a default value, clogging up the metadata.

I wouldn't see it this way. E.g. in nbsphinx I have the setting "allow_errors", which is by default false. And it doesn't have to be present, the metadata can stay clean. If it's not present, the default value kicks in. You can still make the setting explicit with "allow_errors": false. This becomes meaningful once the default value is changed by setting the global option nbsphinx_allow_errors = True. This wouldn't be possible with your proposed tag scheme, right? Or would you have multiple tags? Something like "allow errors" and "disallow errors"?

I cannot evaluate whether it is a good idea for all of the cases in which people would want to use it.

You are right, my statement was too general. I should have said something like "hiding code isn't a good idea most of the times".

There are tonnes of people who have been requesting this feature for an extremely long time.

I'm aware of this. I've gotten those requests myself. I think most of the times, they would be better off not hiding their code.

I'm confident that at least one person could write better documentation for their software using toggleable cells/inputs/outputs than they could without.

That's indeed quite likely. But I haven't seen such a case yet. I'm looking forward to seeing one!

Anyway, I'm not saying that such a feature should never be implemented, I just wanted to tell you why I haven't implemented it.

[mpacer]

Quick comments on my phone, sorry for non quotes, any comments appearing out-of-order and a lack of formatting.

This doesn't need an enhancement proposal it's already in nbformat and on the development path. One of the mandates in the jupyter grant is to make hiding elements in outputs based on tags happen. I have developed my thoughts on this not only from the perspective of nbconvert but in collaboration with @carreau and @fperez. My words are just my attempt at crystallizing this line of reasoning.

Collapsed is a metadata attribute, at the same level as tags. The additive format of tags was designed with the UI question specifically in mind. In particular, it was partially designed with tasks such as quickly marking cells to be hidden on output as one use case (or so I have surmised).

All having the default value as you do does is map the 3 value (2 feature) state system back to a 2 value system. You could make the presence of the default argument have a different value altogether. Look at the implementation of alt text for HTML image tags for an example of a useful version of such a system (no alt tag is different from alt="").

Applying the hide_cell tag does not hide it from view in the notebook, only in the converted document, thus there is no conflict with it being hidden or not. This is why it is appropriate to discuss this from the perspective of nbconvert. Collapsible or hideable live input areas are another feature idea but to the best of my knowledge there is no effort currently being pursued to implement that. If that were the case I imagine there would be discussion about whether there should be a separate UI for that.

You would either have allow errors or disallow errors be the default. You could have the exact same syntax for setting it but change the mechanism to put nothing there if it's false and something there if it's true. But you're also creating that as an intermediate metadata representation for your computations. This means that even if it does become cluttered that's less of an issue than it is for a file format's metadata.

All the tags need to appear as strings in a flat list in the current nbformat so there is no way to have sets of tags. You can have other metadata attributes that have other kinds of structure but they would not be tags as described in the format.

That's fine! You could have just not implemented it because you also have higher priority tasks. So knowing why you chose not to do so is helpful. I greatly appreciate you taking the time to explain your reasoning. It has helped me think about this problem!

[mgeier]

Thanks for the additional info, that's really enlightening!

Applying the hide_cell tag does not hide it from view in the notebook, only in the converted document, thus there is no conflict with it being hidden or not. This is why it is appropriate to discuss this from the perspective of nbconvert.

I wasn't aware of that! I thought the hiding mechanism was meant to be used in both the live Notebook and nbconvert. Now many things make more sense to me!

This is good and bad news to me: It's bad because having no dedicated UI in the live Notebook will make this feature much less accessible to "normal" users. It's good, because IMHO people should use this very sparingly anyway (as mentioned above). Now only those who really want it will use it. Which is good!

With all this cleared up, there is one question left: I don't like the idea of a prefix anyway (as mentioned above), but doesn't the prefix jupyter: suggest that the feature is available for the whole Jupyter system, i.e. also for the live Notebook? If you really have to have a prefix, wouldn't it make more sense to make it more specific to nbconvert, e.g. nbconvert:hide_input?

[ellisonbg]

subscribing...

[ellisonbg]

For the record I am in favor of the "jupyter:" prefix within the tag list of cell metadata. The particular tags we are thinking about in this context should be project wide, rather than notebook of nbconvert specific.

[mpacer]

Just to quickly weigh in: I agree with @ellisonbg about the project wide namespace with project specific semantics (i.e., even if eventually there is a way to hide an input cell in the notebook itself, that should different from jupyter:hide_input such as jupyter:hide_live_input). From what I understood, the idea is that the prefix only indicates a pseudonamespace for tags with official Jupyter support, rather than anything about the semantics of the tags as used by various projects.

I believe the goal is to have a dedicated UI for assigning tags in the notebook (if not in the classic notebook, at least in jupyter-lab). It's just that the meaning of jupyter:hide_input doesn't include hiding cell input within the notebook itself. I think the UI for hiding inputs/outputs/cells in a living notebook does pose an issue (e.g., if you want to unhide those inputs/outputs/cells) but that should be addressed elsewhere.

[Carreau]

Agreed that a prefix is needed. Neutral for jupyter:, anyway in the UI the prefix can be made less-visible like shown only if ambiguous.

[gpagliuca]

As user I'd like to show the output, not hide the input (it is not the same thing).

It would be nice to show some variables calculated in one or more hidden cells at the top of the notebook in a markdown cell. This is already possible in the jupyter notebook thanks to this extension python-markdown.

PROs: no "orphan" cell with only the output, easy to implement (I guess) since no modifications are required upstream in the notebook code, all handled within nbconvert CONs: not sure if this would work with non-python kernels, does not work in LaTeX as it is now

[mpacer]

Hi @gpagliuca I think that we are using slightly different meanings of "output".

When we are saying "hide_output" for a cell, what we're referring to is not the python state due to the cell, but rather the output prompt area that is displayed below the input prompt area.

What it sounds like you want is the ability to reference objects in the python namespace within markdown cells and have their repr() displayed. That is a slightly different issue, and wouldn't be accomplished with cell level tags. We are unlikely to implement that in the core of nbconvert because:

1. (less important) it is already available as an extension (as you point out)
2. (more important) it requires a deviation from most common markdown standards in order to interpret {{…}} in the way that is suggested.

If you want this in nbsphinx output, I think it should work today as long as you have the configuration change required available to the Exporter:

c = get_config()
c.Exporter.preprocessors = ['pre_pymarkdown.PyMarkdownPreprocessor']

I'm surprised it poses a problem with LaTeX exporting though, because it's acting as a preprocessor which should change the notebook before jinja touches anything, meaning that the curly braces in the {{…}} syntax shouldn't cause as many issues as it sounds like it is. What might need to happen is to ensure that it is applied before escaping literal {s and }s.

edit: Ah — it doesn't have a problem with conversion to LaTeX, it has a problem with its activation within a LaTeX maths environment as indicated via $…$ or $$…$$. I see why that would cause an issue.

[jgosmann]

In nbformat 4.4 there are official metadata keys for hiding the input and output.

[mpacer]

@jgosmann The metadata keys that you linked to indicate the current state of the in-browser visualisation of the notebook. These fields are not to be interpreted to mean hidden for export and will not be supported in nbconvert for that purpose. We are working on other ways to remove/hide elements from notebooks see https://github.com/jupyter/nbconvert/pull/640 and https://github.com/jupyter/nbconvert/pull/643 for an example of how to do this based on tags.