Configure notebook pairing using regex/fnmatch/glob patterns

[Jacob-Stevens-Haas]

I have a library with examples in the examples folder. Each example is its own subfolder, containing:

• example.py (want to sync)
• example.ipynb (want to sync)
• utils.py (helper functions for example.py/ipynb - don't sync)

In pyproject.toml, I tried setting:

[tool.jupytext]
formats = "examples/*/example.ipynb///ipynb,examples/*/example.py///py:percent"

However, jupytext would not find & correct my file either with pre-commit run --all or jupytext --pre-commit --sync

Failing the above, I also can't seem to set pairings individually in pyproject.toml or in .pre-commit-config.yaml. For the latter, I tried:

- repo: https://github.com/mwouts/jupytext
  rev: v1.14.0
  hooks:
    - id: jupytext
      args: ["--set-formats", "ipynb,py", "examples/2_introduction_to_sindy/example.py", "--sync"]

But I got the error:

...
jupytext: error: unrecognized arguments: <list of most of the files in my repo>

How do I achieve this?

[abhishekmittal15]

This is something I need too. To be able to specify the config in pyproject.toml only for certain notebooks and not for all notebooks. I see a merge here, but not sure if the merge addressed this issue. Has the issue been resolved @Jacob-Stevens-Haas ?

[Jacob-Stevens-Haas]

@abhishekmittal15 , that PR was on the repository I work on, mentioning that we can't add jupytext as a pre-commit until this issue is resolved.

[mwouts]

Hi @Jacob-Stevens-Haas , sorry for not answering this before. The following did work in my test:

[tool.jupytext]
formats = "examples//example/ipynb,examples//example/py:percent"

The first "examples" is the root folder where you require paired files to be, and the second "example" is the required name prefix.

I do agree that we should give more details about this the documentation, I'll keep a note of that on my todo list.

Regarding your second question, local configuration files do work (and if you don't need to filter the root folder then the syntax is just formats = "example/ipynb,example/py:percent" if you are just willing to filter on an example prefix). I think what was did not work in the attempt documented above is the explicit file name in the args argument of the pre-commit hook.

[Jacob-Stevens-Haas]

Thanks @mwouts, works better, but is there a way to exclude files like examples//example_data.py? All of my example notebooks offload data generation to those modules.

I also get warnings for extraneous files that are not paired:

[jupytext] Reading test/utils/test_axes.py in format py
[jupytext] Warning: test/utils/test_axes.py is not a paired notebook

I tried silencing these with -q, but instead of silencing the warnings, it silenced the errors and left the warnings! i.e.

[jupytext] Updating examples/2_introduction_to_sindy/example_data.ipynb

no longer appears.

Also, I tried to run jupytext outside of pre-commit with jupytext --sync --pre-commit, but is there a way to get the effect of pre-commit run -all? In the interim, I've made some test changes to the repo so I can enter the debugger and see how pairs.py and config.py are working.

[Jacob-Stevens-Haas]

FWIW, I think the DeprecationWarning here should be a UserWarning, since DeprecationWarning is ignored by default unless raised in __main__ module. If I understand correctly, DeprecationWarning would make sense if client libraries were using cli.py.

[mwouts]

Hello @Jacob-Stevens-Haas , thank you for your previous remarks

is there a way to exclude files like examples//example_data.py?

No, there is no way to do so. The filtering is only on the folder / prefix or suffix (and I don't have the bandwidth to change that)

I tried silencing these with -q, but instead of silencing the warnings, it silenced the errors and left the warnings!

The warning on unpaired files was also noticed at #986. Maybe we could remove that warning and simply issue a warning if --sync is used but no paired file is found among the arguments... What do you think?

I tried to run jupytext outside of pre-commit with jupytext --sync --pre-commit, but is there a way to get the effect of pre-commit run -all?

Yes, as you probably found out, it's --pre-commit-mode, cf. https://github.com/mwouts/jupytext/blob/12964ef42783722815cd5d12df47d5ce67c1b323/.pre-commit-hooks.yaml#L5

I think the DeprecationWarning here should be a UserWarning,

Do you mean the warning does not appear at the command line? If so yes sure a UserWarning would be a better fit, certainly.