Notebook validation failed: Non-unique cell id

[kohlerjl]

I have recently started receiving this popup error frequently when saving notebooks:

Title: Notebook validation failed

The save operation succeeded, but the notebook does not appear to be valid. The validation error was:
-------------------------
Notebook validation failed: Non-unique cell id 'waiting-opening' detected. Corrected to 'noted-romania'.:
"<UNKNOWN>"

It does not occur in new notebooks, but seems to be triggered after copying and pasting cells from other notebooks. But it is now occurring frequently on two different systems, editing different sets of notebooks. Once it appears within a notebook, it reappears on every subsequent save, with the same 'non-unique cell id' detected, but a different 'corrected to ' value.

I've attached the full output of conda list, but the versions of some particularly relevant packages are:

jupyter 1.0.0 py37h03978a9_6 conda-forge jupyter_client 6.1.11 pyhd8ed1ab_1 conda-forge jupyter_console 6.2.0 py_0 conda-forge jupyter_contrib_core 0.3.3 py_2 conda-forge jupyter_contrib_nbextensions 0.5.1 py37hc8dfbb8_1 conda-forge jupyter_core 4.7.1 py37h03978a9_0 conda-forge jupyter_highlight_selected_word 0.2.0 py37h03978a9_1002 conda-forge jupyter_latex_envs 1.4.6 py37hc8dfbb8_1001 conda-forge jupyter_nbextensions_configurator 0.4.1 py37h03978a9_2 conda-forge jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge nbconvert 5.6.1 py37hc8dfbb8_1 conda-forge nbformat 5.1.2 pyhd8ed1ab_1 conda-forge notebook 6.2.0 py37h03978a9_0 conda-forge

[kohlerjl]

conda_list.txt

[kevin-bates]

This message is related to some fairly recent changes to nbformat that introduce cell id metadata for each notebook cell. In this case, the validation logic is encountering a duplicate cell-id that was previously, and randomly, generated 'waiting-opening' and encountering that same cell-id later in the notebook. It's strange that this occurs for the same cell-id value and is as if the corrected-to value is not getting persisted. I'm unable to reproduce this, but I suspect there are a few factors in play here.

I'm cc-ing @MSeal for comment as to what might be going on and how best to proceed (as I'd rather not recommend downgrading).

[MSeal]

Sorry I accidentally hit the close button. 👀 on this now

[MSeal]

I also am struggling to reproduce the event with a very similar dependency list. Even if I manually force cell ids to be invalid / equal the notebook server corrects it with version 6.2.0 before it ever get to the validation error reported here.

@kohlerjl could you post a notebook exhibiting the behavior? I am wondering if something in the file structure will lend a clue as to why the replaced cell-id is not being fixed.

[kohlerjl]

Thanks for the quick follow up.

I spent some time trying to isolate a simple notebook that would reproduce the event. What I've found is that this behavior does not persist between closing and reopening the notebook, or even just refreshing the browser tab (leaving the kernel still running).

However, I've been able to consistently reproduce this behavior by following these steps:

1. Create a new notebook and enter some code in a cell (i.e. 'a = 1')
2. Save the notebook, then refresh the tab
3. Copy the cell and paste a duplicate within the same notebook
4. Save the notebook again, triggering the error

It appears that this behavior does not occur if you duplicate a cell created in the same 'session' (i.e. while the notebook is open in the tab). But If I copy and paste a cell created prior to opening/refreshing the notebook, either from the same notebook or a different notebook, then I get errors about duplicate ids. But the notebook changes do save, and I can simply refresh the tab to mitigate the errors.

I have attached a notebook I produced this way, which gave me the errors prior to reloading it. However, I can't see any difference in the file structure. test.ipynb.txt

I also cannot reproduce this behavior on another system, running Python 3.9.2 and: ipython 7.19.0 jupyter-client 6.1.7 jupyter-console 6.2.0 jupyter-core 4.6.3 jupyter-server 1.4.1 nbconvert 6.0.7 nbformat 5.0.8 notebook 6.2.0

[kevin-bates]

Thanks @kohlerjl. I'm still unable to reproduce this (sorry). Are you using the Notebook classic or Juptyer Lab interface? (Neither reproduces the issue for me however.)

I also cannot reproduce this behavior on another system, running Python 3.9.2 and: ... nbformat 5.0.8

This would be because that version of nbformat doesn't contain this consistency check. I'm using nbformat 5.1.2, but, again, no luck. Really curious what else could be going on.

Btw, we started displaying the notebook server version in the console when the server is started in the 6.x timeframe. Could you please confirm the displayed value? The log entry should look similar to the following:

[I 10:06:42.433 NotebookApp] Jupyter Notebook 6.2.0 is running at:

[kohlerjl]

I can confirm that the notebook server reports "Jupyter Notebook 6.2.0 is running at:" on startup.

I've been using Notebook classic. I installed Jupyter Lab and tried the same procedure, but cannot reproduce the behavior there.

I might just transition to using Jupyter Lab going forward, since that seems to be more actively developed now.

[kevin-bates]

I can confirm that the notebook server reports "Jupyter Notebook 6.2.0 is running at:" on startup.

Thank you. I'll have to defer to @MSeal on this one.

I've been using Notebook classic. I installed Jupyter Lab and tried the same procedure, but cannot reproduce the behavior there.

Just a point of clarity, Jupyter Lab >= 3 uses a different server (jupyter_server) whereas Lab < 3 still uses notebook as its server - although I believe the issue here lies in the front-end where cells are manipulated. As a result, it might be helpful to know which version of Lab did not reproduce this behavior and whether you've tried this with Lab < 3.

... Jupyter Lab ... seems to be more actively developed now.

Yes, that is absolutely the case.

[njohnsson]

I get a similar error saying cell ID was corrected to 'domestic-communist'. From comments above it seems cell-id names are randomly generated. Maybe that randomization algorithm should be changed a bit. In this case I thought I was dealing with a virus...

Title: Notebook validation failed The save operation succeeded, but the notebook does not appear to be valid. The validation error was: Notebook validation failed: Non-unique cell id 'moving-ultimate' detected. Corrected to 'domestic-communist'.: "<UNKNOWN>"

I am running: Classic Notebook (Not Jupyter Lab), v 6.2.0; Ubuntu; Chrome Browser Python 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05)

[MSeal]

@njohnsson make sure you have the latest package versions in your environment. We changed the id algorithm to use hashes and invalidated some of the older packages with name based id generation because it was creating problematic ids and marked the older nbformat packages as deprecated.

Similar to @kevin-bates I've struggled to reproduce the issue in classic. That being said classic is not being actively developed. If you're looking for the same simple look-n-feel I'd suggest using Retrolab or NBClassic with lab instead as new package capabilities will slowly be less supported in classic over time.

[njohnsson]

@MSeal: OK, I will update package versions, but just FYI: I the latest message I got ended with "....Non-unique cell id 'civil-spring' detected. Corrected to 'lesbian-voluntary'."

[EricThomson]

I have been getting this as I do a lot of multi-cell ctrl-c/ctrl-v in my standard jupyter notebook lately (not jupyterlab). I am in jupyter_client 6.1.12 in Windows 10 installed using conda, working in firefox with Pytyon 3.7.6.

It seems to go away when I close/reopen the nb.

[MSeal]

@njohnsson You can see why we revoked the nbformat packages. It was horribly problematic and attempts to correct the lexicon where still showing problematic combinations. The new version (5.1.3) just uses hashes.

@EricThomson FYI the change needed is around nbformat and notebook server. The jupyter_client package is mostly unrelated.

[hadivafaii]

This worked for me as a temporary fix:

import nbformat as nbf
from glob import glob

import uuid
def get_cell_id(id_length=8):
    return uuid.uuid4().hex[:id_length]

# your notebook name/keyword
nb_name = 'my_notebook'
notebooks = list(filter(lambda x: nb_name in x, glob("./*.ipynb", recursive=True)))

# iterate over notebooks
for ipath in sorted(notebooks):
    # load notebook
    ntbk = nbf.read(ipath, nbf.NO_CONVERT)
    
    cell_ids = []
    for cell in ntbk.cells:
        cell_ids.append(cell['id'])

    # reset cell ids if there are duplicates
    if not len(cell_ids) == len(set(cell_ids)): 
        for cell in ntbk.cells:
            cell['id'] = get_cell_id()

        nbf.write(ntbk, ipath)

[irmagaladi]

This worked for me as a temporary fix:

import nbformat as nbf
from glob import glob

import uuid
def get_cell_id(id_length=8):
    return uuid.uuid4().hex[:id_length]

# your notebook name/keyword
nb_name = 'my_notebook'
notebooks = list(filter(lambda x: nb_name in x, glob("./*.ipynb", recursive=True)))

# iterate over notebooks
for ipath in sorted(notebooks):
    # load notebook
    ntbk = nbf.read(ipath, nbf.NO_CONVERT)
    
    cell_ids = []
    for cell in ntbk.cells:
        cell_ids.append(cell['id'])

    # reset cell ids if there are duplicates
    if not len(cell_ids) == len(set(cell_ids)): 
        for cell in ntbk.cells:
            cell['id'] = get_cell_id()

    nbf.write(ntbk, ipath)

Also for me! Thank you very much for sharing. I cannot copy-paste cells in my jupyter notebook... because this error always appears afterwards. But with this code, the error disappears. After running it, what worked for me is to save and on the message that appears on a new windows click on "Reload".

[aloosley]

I have been getting this as I do a lot of multi-cell ctrl-c/ctrl-v in my standard jupyter notebook lately (not jupyterlab). I am in jupyter_client 6.1.12 in Windows 10 installed using conda, working in firefox with Pytyon 3.7.6.

It seems to go away when I close/reopen the nb.

I appear to have the same issue after copying / pasting cells (pop_os! 20.10, python 3.9.5, jupyter 1.0.0). Thanks for the fix everybody.

[EricThomson]

Workaround I've been using: cut in command mode (blue margin) but paste in edit mode (green margin).

No more errors.

Downside: when you paste in edit mode it all gets thrown into one cell, and it is put in code mode, so if you have a ton of formatted cells with lots of markdown, you will have to redo that). For my use case it is not that big of a deal so I'm pretty happy with this workaround.

[shankari]

@MSeal I am still seeing name-based cell ids, even on nbformat 5.1.3.

Background:

• I have a repository with some notebooks checked in.
• The notebooks don't have outputs embedded.
• My goal is to be able to diff the notebooks and review them and to follow a standard source control format

Since around March of this year, the "cell id" has been causing problems with this approach, since when I use "Reset kernal and Clear Outputs" to clear my outputs before checking in, all the ids change. This makes it really hard to identify the real changes.

I found this PR today, so I upgraded to nbformat 5.1.3

Downloading and Extracting Packages
folium-0.12.0        | 64 KB     | ################################################################### | 100%
nbformat-5.1.3       | 47 KB     | ################################################################### | 100%
...

$ conda list | grep nb
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libopenblas               0.3.12          openmp_h54245bb_1    conda-forge
nbclient                  0.5.1                      py_0    conda-forge
nbconvert                 6.0.7            py37hf985489_3    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
widgetsnbextension        3.5.1            py37hf985489_4    conda-forge

I then ran "Kernel -> Restart and Clear Output". The new ids generated are still name-based and don't appear to be hashes - e.g.

-   "id": "cutting-april",
+   "id": "furnished-webcam",
    "metadata": {},
    "source": [
     "### Read data and setup variables"
@@ -119,7 +131,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "external-westminster",
+   "id": "french-place",

Is this expected?

My project is open source, so I have an environment and notebooks that I can share with you. But it seems like what you really need are logs. Happy to send you as many as you like if you let me know where to get them.

This is currently 100% reproducible for me.

[shankari]

Good news: I created a new notebook instead of editing an existing one, and now the IDs do seem to be hashes! Bad news: The ids still seem to change although there are no changes in the cell

See https://github.com/e-mission/e-mission-eval-private-data/pull/28/commits/d81a23417a9cd7ca5d3a2abac561c46976ae778e for an example. There are no changes in the several of the cells, but the hashes have changed.

[ginward]

I am having similar issues ...

[chivalry123]

any solution yet?

[SoundBoySelecta]

Similar issues, the only action I was performing was copying a horizontal line

---

and pasting it else where, It has something to do with what ET mentioned... first time I copied the horizontal line in command (blue) mode, and then pasted without creating a new cell, so it was pasted in command mode when a cell was highlighted (existing empty cell), the second time I copied it but pasted the markdown into a cell in edit more (green). Wasn't able to replicate it in a new page. Whats also interesting (not sure if intentional), when I paste the horizontal like in command mode, vs the markdown code into a cell in edit mode, the colour of the lines are two different shades.

[SoundBoySelecta]

I used download as, saved file as a .ipynb, reopened with no errors. One method I was thinking of if the error persists is import the raw format which is a dict of dict and write some code to get all "id" and get the duplicate id, count the index position of the duplicate "id", then delete that cell in either the nb or the dict of dicts.

[JMBurley]

I just updated my notebook packages yesterday and started getting this error when I copy paste anything (perhaps because I am working with notebooks made on older versions on jupyter notebook).

It is pretty crippling to workflow. Any copy-paste of a cell means you have to close and reopen the notebook to stop the save warnings popping up every few mins. Not ideal.

[JMBurley]

@kevin-bates @MSeal Is this issue extant and does it have a path to solution?

@hadivafaii posted a solution that could be a fallback in the save routine to stop this problem ever reaching the end user.

[Cpetro02]

+1 also still having this problem

[MSeal]

The solution has to be done in the frontend here of Jupyter. I don't maintain any of the JS code here but the path @hadivafaii makes should be how the save operation need to change in JS side of things.

[jasongrout]

FYI, here is the issue and pull request we made in jlab for supporting cell ids.

[blois]

Is this related to https://github.com/jupyter/notebook/pull/5928? It should be clearing the ID before putting the item on the clipboard but appears that something is going wrong there.

I'm having trouble repro'ing, but would it make sense to revert https://github.com/jupyter/notebook/pull/5928? Being unable to save is worse than unstable cell IDs. Though there may just be an edge case which was missed in the PR and can be cleaned up.

[MSeal]

I not positive reverting cell id awareness would resolve the issue. Also it causes every notebook save to replace ids if we revert it which caused a lot of personal DMs to me about git diffs generated from other projects :joy: .

@blois I can reproduce but I sometimes had to refresh the page twice or restart the server to get it to have the duplicate cell id. I'm not sure why local state was affecting it. Is there a second path for copy where it wouldn't clear the cell id in the buffer?