README file in a parent directory raises error "Crucial files were excluded from the sdist"

[frankie567]

Problem

I have a mono-repository containing several Python libraries. Each one has its own build configuration with its own pyproject.toml file. However, I have only one README file which I want to be common for each one. Here is what the directory structure looks like:

.
└── repository/
    ├── libraries/
    │   ├── lib1/
    │   │   └── pyproject.toml
    │   ├── lib2/
    │   │   └── pyproject.toml
    │   └── lib3/
    │       └── pyproject.toml
    └── README.md

So what I did in my pyproject.toml files is to set for the description-file option a relative path to the README file:

description-file = "../../README.md"

However, when triggering the build, here is what I get:

Found 50 files tracked in git                                                                                                                                                                                                     I-flit.sdist
Traceback (most recent call last):
  File "/Users/fvoron/Development/fastapi-users/libraries/fastapi-users/venv/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/fvoron/Development/fastapi-users/libraries/fastapi-users/venv/lib/python3.8/site-packages/flit/__init__.py", line 154, in main
    main(args.ini_file, formats=set(args.format or []),
  File "/Users/fvoron/Development/fastapi-users/libraries/fastapi-users/venv/lib/python3.8/site-packages/flit/build.py", line 46, in main
    sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
  File "/Users/fvoron/Development/fastapi-users/libraries/fastapi-users/venv/lib/python3.8/site-packages/flit_core/sdist.py", line 171, in build
    files_to_add = self.apply_includes_excludes(self.select_files())
  File "/Users/fvoron/Development/fastapi-users/libraries/fastapi-users/venv/lib/python3.8/site-packages/flit_core/sdist.py", line 146, in apply_includes_excludes
    raise Exception("Crucial files were excluded from the sdist: {}"
Exception: Crucial files were excluded from the sdist: ../../README.md

From what I understand, some code here checks that every file has been included in the package. I guess that since my README file is in a parent folder, it's not included.

However, as per the description_from_file function it seems that Flit is perfectly able to read the content of the file.

What I tried

• Create a symbolic link pointing to the README file inside the library folder (next to pyproject.toml). It gives an error; I guess symbolic links are ignored at some point:

Found 50 files tracked in git                                                                                                                                                                                                     I-flit.sdist
Writing generated setup.py                                                                                                                                                                                                        I-flit.sdist
Built sdist: dist/fastapi-users-6.1.2.tar.gz                                                                                                                                                                                 I-flit_core.sdist
Config error: Description file /var/folders/c0/qb_x1mf528d5n05l3zpg7f_00000gn/T/tmp_s4aygiq/fastapi-users-6.1.2/README.md does not exist

• Create a copy of the file before running the build, but triggers the Git dirty check ("Untracked or deleted files in the source directory. Commit, undo or ignore these files in your VCS.").

Questions

• Is this a case that should be supported by Flit? If yes, I would be happy to help with the implementation.
• If not, any ideas for a workaround (apart from having duplicate README files)?

[sgbaird]

Running into something similar:

(base) PS C:\Users\sterg\Documents\GitHub\sparks-baird\mat_discover\mat_discover\CrabNet> flit publish
Adding generated setup.py to sdist by default.                  I-flit
This will change in a future version. Pass --[no-]setup-py to control it.  I-flit
Found 724 files tracked in git                            I-flit.sdist
Traceback (most recent call last):
  File "C:\Users\sterg\Miniconda3\Scripts\flit-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\Users\sterg\Miniconda3\lib\site-packages\flit\__init__.py", line 191, in main
    main(args.ini_file, repository, args.pypirc, formats=set(args.format or []),
  File "C:\Users\sterg\Miniconda3\lib\site-packages\flit\upload.py", line 265, in main
    built = build.main(ini_path, formats=formats, gen_setup_py=gen_setup_py)
  File "C:\Users\sterg\Miniconda3\lib\site-packages\flit\build.py", line 46, in main
    sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
  File "C:\Users\sterg\Miniconda3\lib\site-packages\flit_core\sdist.py", line 171, in build
    files_to_add = self.apply_includes_excludes(self.select_files())
  File "C:\Users\sterg\Miniconda3\lib\site-packages\flit_core\sdist.py", line 146, in apply_includes_excludes 
    raise Exception("Crucial files were excluded from the sdist: {}"
Exception: Crucial files were excluded from the sdist: CrabNet\__init__.py

Doesn't seem to matter if I delete CrabNet/__init__.py or not.

[merwok]

It’s not clear if you’re seeing the same thing or not. The original post was well defined: README file in parent directory. Can you tell more details about your project layout? We don’t know if CrabNet is the project top dir or the Python importable package dir (in which case it should contain __init__.py) or something else.

[SilverBzH]

Hey, do you have any update regarding the original issue ?

[takluyver]

I don't think there's a good way for a README in a parent directory to work. The pyproject.toml goes into the sdist unmodified (it would be a significant extra complexity for Flit to start rewriting it), and we can't put something in a parent directory inside the sdist.

So maybe the solution here is just to fail sooner with a clearer error (e.g. 'Readme file cannot be in a parent directory').

[SilverBzH]

I'm currently doing a workaround, I've added the parent folder .. into [tool.flit.stdist].

With this configuration I am able to build with python3 -m build, which is "okay" for now. But Flit's build is crashing because of this assert. I haven't dig into the code so I don't know if this assert make sense or not.

Currently I think moving the README.md from the parent folder to the child folder might be the best idea. Do you have any plan to support parent folder ?