config.find_juptext_configuration_file('.') doesn't find jupytext.toml in a parent directory - Fix suggested

[ryan-feeley]

Hi, I'm seeing the following issue on Windows 10, python 3.8. After reviewing the source code, I expect it applies to all OSs and versions of Python3.

When I run these three lines, the second one returns empty, when there is in fact a jupytext.toml file in the parent folder. That is confirmed by the third line.

from jupytext.config import find_jupytext_configuration_file
find_jupytext_configuration_file('.')
find_jupytext_configuration_file('..')

It appears the issue is related to line 339 of config.py, which returns empty when path == '.'.

parent_dir = os.path.dirname(path)

I don't know enough about os or python across platforms (I usually use pathlib) so be sure of a pull request. But I think the following would fix my issue:

parent_dir = os.path.dirname(os.path.abspath(path))

I'm surprised that os.path acts this way on my system, but it does. If you call os.path.split('.'), you will also get an empty string for the parent. So you need to first expand the string '.' before operating on it.

Looking at the os documentation, I guess this is expected:

os.path.dirname(path):

Return the directory name of pathname path. This is the **first element of the pair returned by
passing path to the function split()**.

os.path.split(path):

Split the pathname path into a pair, (head, tail) where tail is the last pathname component and
head is everything leading up to that. The tail part will never contain a slash; if path ends in a
slash, tail will be empty. **If there is no slash in path, head will be empty**.

[ryan-feeley]

Just for some context, I came across this issue while trying to figure out why a jupytext.toml file in the parent directory wasn't causing the .py to be automatically generated. My above edit to config.py allows find_jupytext_configuration_file('.') to list the correct config file from the parent directory, but it still isn't being used in my jupyter notebook.

I don't have anything set for JUPYTEXT_CEILING_DIRECTORIES, so I would expect stuff to work with the config file in the parent directory, but it doesn't appear to.

[mwouts]

Hello @ryan-feeley , thank you for reporting this!

Well there is a lot to comment on here... Maybe I'll start with what looks really problematic here:

I came across this issue while trying to figure out why a jupytext.toml file in the parent directory wasn't causing the .py to be automatically generated

I guess this happens in Jupyter, is that correct? Can you check that your configuration file is below the root directory for Jupyter (--notebook-dir)? In other words, can you access the configuration file from Jupyter? If not, Jupytext won't be able to see it neither (but it will still be able to see global configuration files)

I usually use pathlib

Oh yes that would be a great idea! Maybe I'll do the switch at some point in the future. For now I have created a branch but I am afraid that just switching to pathlib will not solve the issue, because in pathlib as well we have Path('.').parent == Path('.').

parent_dir = os.path.dirname(os.path.abspath(path))

Yes indeed taking the absolute path seems to be a way to get a different parent for '.'. However I am not sure we should expand all paths like this, and maybe the fact that find_jupytext_configuration_file('.') does not find your configuration file is not related to what happens in Jupyter, cf. my first comment above. So maybe I'll prefer to know a bit more about the location of the config file, relative to the root of your Jupyter server?

[hectormz]

I was just about to open an issue about this.

I often have a project structure as follows:

.
├── .git
├── jupytext.toml
└── notebooks
    ├── demo.ipynb
    └── demo.py

If I launch Jupyter Lab in my notebooks folder (preferred), jupytext does not find my config files, so I have to launch from project root where the config files are. Should I be able to launch from notebooks and have jupytext find the config files in parent directories?

I'm also on Windows with Python 3.8/3.9

[mwouts]

Hi @hectormz , thank you for following up on this.

If you launch in notebooks, then as long as we are using the get method of the Jupyter contents manager, then we cannot access the configuration file that resides above that folder.

I have a preference for using the get method rather than accessing files directly, as some people use specific contents managers like e.g. jupyter-fs.

Of course we could document that limitation in the documentation - I'll be happy to make or take a PR if you think this is the appropriate answer to this issue.

Also, did you have a look at the other option, which is to use a global configuration file?

[hectormz]

Hi @mwouts thanks for the information and clarification. Could you provide more information (link) about the get method from Jupyter contents manager? I'm a bit busy to make a PR for the clarification in documentation, but if you can wait a week, I would be happy to do it!

At the moment I would like to avoid a global configuration file because I may be sharing a repository with others, and want to ensure we have the same behavior. To complicate things slightly, I actually have been using the following structure, so that my notebooks folder is only .py files:

.
├── .git
├── jupytext.toml
├── nb_ipynb
|   └── demo.ipynb
└── notebooks
    └── demo.py

I like having configuration files at the root of a project anyway, but I think I had to move the jupytext configuration to make it sync to the other folder.

[mwouts]

Thank you @hectormz for your understanding. Sure a documentation PR would be great! (and there is no hurry at all)

The get method of Jupytext's contents manager is inherited from the parent content manager, and has the following signature:

https://github.com/jupyter/notebook/blob/ccd9665571107e02a325a738b8baebd6532b2d3d/notebook/services/contents/manager.py#L249-L251

(of course for most users the implementation of get defaults to a simple file read, cf. here).

The path argument is relative to the folder in which Jupyter is started, so that's why we cannot access configuration files above the Jupyter root dir with the get method (or maybe we can, actually I did not try, but that would sound a bit hacky anyway, don't you think?)

[Aetf]

Not sure if this is related, but I feel it's similar. I'm using jupytext in the command line and the config file in the parent folder won't take effect unless I run jupytext from the parent folder directly.

Here are the steps:

❯ ls --tree
./
├── __notebooks__/
│  └── demo.ipynb
└── jupytext.toml
❯ jupytext --sync __notebooks__/demo.ipynb
[jupytext] Reading __notebooks__/demo.ipynb in format ipynb
[jupytext] Updating ./demo.py
❯ ls
__notebooks__/  demo.py  jupytext.toml
❯ rm demo.py
❯ cd __notebooks__
❯ jupytext --sync demo.ipynb
[jupytext] Reading demo.ipynb in format ipynb
[jupytext] Warning: demo.ipynb is not a paired notebook
❯ ls
demo.ipynb
❯ ls ..
__notebooks__/  jupytext.toml

The content of jupytext.toml:

formats = [
    "__notebooks__///ipynb",
    ".///py:percent"
]