Intermitent failure: `test_ipynb_notebooks_can_be_trusted[ipynb_py/frozen_cell.ipynb-async]`

[mwouts]

I see intermittent failures on the CI like e.g. here:

=================================== FAILURES ===================================
____ test_ipynb_notebooks_can_be_trusted[ipynb_py/frozen_cell.ipynb-async] _____
[gw1] darwin -- Python 3.13.2 /Users/runner/miniconda3/envs/jupytext-ci/bin/python

ipynb_py_file = '/Users/runner/work/jupytext/jupytext/tests/data/notebooks/inputs/ipynb_py/frozen_cell.ipynb'
tmpdir = local('/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/pytest-of-runner/pytest-0/popen-gw1/test_ipynb_notebooks_can_be_tr62')
no_jupytext_version_number = None
cm = <jupytext.async_contentsmanager.build_async_jupytext_contents_manager_class.<locals>.AsyncJupytextContentsManager object at 0x10a3a1270>

    async def test_ipynb_notebooks_can_be_trusted(
        ipynb_py_file, tmpdir, no_jupytext_version_number, cm
    ):
        if "hash sign" in ipynb_py_file:
            pytest.skip()
        root, file = os.path.split(ipynb_py_file)
        tmp_ipynb = str(tmpdir.join(file))
        py_file = file.replace(".ipynb", ".py")
        tmp_py = str(tmpdir.join(py_file))
        shutil.copy(ipynb_py_file, tmp_ipynb)
    
        cm.formats = "ipynb,py"
        cm.root_dir = str(tmpdir)
        model = await ensure_async(cm.get(file))
        await ensure_async(cm.save(model, py_file))
    
        # Unsign and test notebook
        nb = model["content"]
        for cell in nb.cells:
            cell.metadata.pop("trusted", True)
    
        cm.notary.unsign(nb)
    
        model = await ensure_async(cm.get(file))
        for cell in model["content"].cells:
>           assert (
                "trusted" not in cell.metadata
                or not cell.metadata["trusted"]
                or not cell.outputs
            )
E           assert ('trusted' not in {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 'start_time': '2018-10-17T10:31:56.157308Z'}, 'trusted': True} or not True or not [{'name': 'stdout', 'output_type': 'stream', 'text': "I'm a regular cell so I run and print!\n"}])
E            +  where {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 'start_time': '2018-10-17T10:31:56.157308Z'}, 'trusted': True} = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 's...print!\n"}], 'source': '# This is an unfrozen cell. Works as usual.\nprint("I\'m a regular cell so I run and print!")'}.metadata
E            +  and   [{'name': 'stdout', 'output_type': 'stream', 'text': "I'm a regular cell so I run and print!\n"}] = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 's...print!\n"}], 'source': '# This is an unfrozen cell. Works as usual.\nprint("I\'m a regular cell so I run and print!")'}.outputs

tests/functional/others/test_trust_notebook.py:54: AssertionError
=============================== warnings summary ===============================
tests/external/jupyter_fs/test_jupyter_fs.py::test_config_jupytext_jupyter_fs_meta_manager[async]
  /Users/runner/miniconda3/envs/jupytext-ci/lib/python3.13/site-packages/traitlets/traitlets.py:3615: DeprecationWarning: metadata {'per_key_traits': <traitlets.traitlets.Dict object at 0x10a424b50>} was set from the constructor. With traitlets 4.1, metadata should be set using the .tag() method, e.g., Int().tag(key1='value1', key2='value2')
    super().__init__(trait=trait, default_value=default_value, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform darwin, python 3.13.2-final-0 ----------
Coverage XML written to file coverage.xml

=========================== short test summary info ============================
FAILED tests/functional/others/test_trust_notebook.py::test_ipynb_notebooks_can_be_trusted[ipynb_py/frozen_cell.ipynb-async] - assert ('trusted' not in {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 'start_time': '2018-10-17T10:31:56.157308Z'}, 'trusted': True} or not True or not [{'name': 'stdout', 'output_type': 'stream', 'text': "I'm a regular cell so I run and print!\n"}])
 +  where {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 'start_time': '2018-10-17T10:31:56.157308Z'}, 'trusted': True} = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 's...print!\n"}], 'source': '# This is an unfrozen cell. Works as usual.\nprint("I\'m a regular cell so I run and print!")'}.metadata
 +  and   [{'name': 'stdout', 'output_type': 'stream', 'text': "I'm a regular cell so I run and print!\n"}] = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'ExecuteTime': {'end_time': '2018-10-17T10:31:56.160823Z', 's...print!\n"}], 'source': '# This is an unfrozen cell. Works as usual.\nprint("I\'m a regular cell so I run and print!")'}.outputs
====== 1 failed, 3794 passed, 204 skipped, 1 warning in 91.30s (0:01:31) =======

[mwouts]

@Darshan808 I see these errors on the CI from time to time since we merged the async contents manager - do you see anything that could be different re trusting notebooks between sync and async contents manager ? Thanks

[mwouts]

For the records there is another occurrence here: https://github.com/mwouts/jupytext/actions/runs/13697163074/job/38302270749

FAILED tests/functional/others/test_trust_notebook.py::test_ipynb_notebooks_can_be_trusted_even_with_metadata_filter[ipynb_py/plotly_graphs.ipynb-sync] - AssertionError: assert False
 +  where False = <built-in method get of NotebookNode object at 0x7f1547569ef0>('trusted', True)
 +    where <built-in method get of NotebookNode object at 0x7f1547569ef0> = {'trusted': False}.get
 +      where {'trusted': False} = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'trusted': False}, 'outputs': [{'data': {'text/html': '      ...tput_type': 'display_data'}], 'source': 'import plotly.offline as offline\noffline.init_notebook_mode(connected=True)'}.metadata
FAILED tests/functional/others/test_trust_notebook.py::test_ipynb_notebooks_can_be_trusted_even_with_metadata_filter[ipynb_py/jupyter.ipynb-async] - AssertionError: assert False
 +  where False = <built-in method get of NotebookNode object at 0x7f1542e7a710>('trusted', True)
 +    where <built-in method get of NotebookNode object at 0x7f1542e7a710> = {'trusted': False}.get
 +      where {'trusted': False} = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'trusted': False}, 'outputs': [{'data': {'text/plain': '3'}, 'execution_count': 1, 'metadata': {}, 'output_type': 'execute_result'}], 'source': 'a = 1\nb = 2\na + b'}.metadata
FAILED tests/functional/others/test_trust_notebook.py::test_ipynb_notebooks_can_be_trusted[ipynb_py/plotly_graphs.ipynb-async] - assert ('trusted' not in {'trusted': True} or not True or not [{'data': {'text/html': '        <script type="text/javascript">\n        window.PlotlyConfig = {MathJaxConfig: \'loca...lotly = Plotly;\n        });\n        }\n        </script>\n        '}, 'metadata': {}, 'output_type': 'display_data'}])
 +  where {'trusted': True} = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'trusted': True}, 'outputs': [{'data': {'text/html': '       ...tput_type': 'display_data'}], 'source': 'import plotly.offline as offline\noffline.init_notebook_mode(connected=True)'}.metadata
 +  and   [{'data': {'text/html': '        <script type="text/javascript">\n        window.PlotlyConfig = {MathJaxConfig: \'loca...lotly = Plotly;\n        });\n        }\n        </script>\n        '}, 'metadata': {}, 'output_type': 'display_data'}] = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'trusted': True}, 'outputs': [{'data': {'text/html': '       ...tput_type': 'display_data'}], 'source': 'import plotly.offline as offline\noffline.init_notebook_mode(connected=True)'}.outputs

[Darshan808]

This issue does not seem to be reproducible. Looks like the unsigning changes are not always being persisted properly before the reload here:

cm.notary.unsign(nb)
model = await ensure_async(cm.get(file))

Not sure about this approach, but since this test is not failing every time and only sometimes, it might be due to timing issues, so adding a small delay between those lines might help? Will let you know if I find anything.

[mwouts]

Oddly enough adding await asyncio.sleep(sleep) seems to help reproduce the issue locally 🤔

pytest tests/functional/others/test_trust_notebook.py::test_ipynb_notebooks_can_be_trusted -n 6

Without -n 6 the tests don't fail, with I have random failures but they are frequent enough.

Also I wondered whether it was fine to save the notebook before popping out the "trusted" metadata or if we needed to do that at all before un-signing... that part dates back a bit!

[mwouts]

(and locally, the failures all occur with sleep > 0!)