pip installation overwrites module directory

[mfdgroot]

There seems to be an error when installing openfermion with pip into a specific target dir. When running pip install -t target openfermion, I get WARNING: Target directory /xxx/target/openfermion already exists. Specify --upgrade to force replacement.. This happens toward the end of the installation process. This is what's in the dir without running with --upgrade and getting the warning:

$ls target/openfermion
chem      _compat.py       config.py    contrib      hamiltonians  linalg        ops          py.typed            testing     utils
circuits  _compat_test.py  conftest.py  functionals  __init__.py   measurements  __pycache__  resource_estimates  transforms  _version.py

And this is after running with --upgrade and without warning:

$ls target/openfermion
docs

So all of the module files are also gone.

[fdmalone]

This looks like a common problem with pip: https://github.com/pypa/pip/issues/8063 that hasn't been resolved. @pavoljuhas have you encountered anything like this before?

[forget-the-bright]

已经被这个折磨一天了，最终选择折中的方法 ，使用pipx 我在下面记录我自己探索解决方法的过程 [Having been tortured by this all day, I finally chose the compromise and used pipx Below I document my own search for a solution] https://memos.haowuan.top/m/gUAJKSzdjEjJZk3grRWL8C

[fdmalone]

going to close this an issue with pip, feel free to reopen if there's another solution.

[pavoljuhas]

This looks like a common problem with pip: pypa/pip#8063 that hasn't been resolved. @pavoljuhas have you encountered anything like this before?

This seems to happen because openfermion declares documentation files as data_files and as such they would install under PREFIX/openfermion. The package sources go to PREFIX/lib/pythonX.Y/site-packages/openfermion. The --target=TARGET option makes pip install both the docs and sources to TARGET/openfermion and it seems only the documentation files survive.

I can think of two remedies

(1) use pip install --prefix=/path/to/prefix instead of --target=TARGET so that the openfermion directories for docs and package sources are different

OR

(2) update openfermion/setup.py to have documentation files included as package_data so they get installed somewhere under .../site-packages/openfermion instead of to PREFIX/openfermion.

[fdmalone]

thanks @pavoljuhas I think 2 is probably best.

[mhucka]

@mfdgroot Thank you for the report.

OpenFermion was doing something unusual: as @pavoljuhas noted, it included documentation source files (.md and tutorial .ipynb files) in the package distribution using the data_files component, and as a result, pip install wrote them to the data files installation destination (which is different from the Python package destination). Those are the only things that the OpenFermion installation would write in that data files destination directory:

I suspect that users never found the files in that location because it's so unusual. Given that the documentation is online at https://quantumai.google/openfermion, and in the worst case, people can find the files in the GitHub repo, the consensus after internal discussions is that it's okay to remove data_files from setup.py and not include these files in the Python package installation.

This change is now done in PR #945 and will be in the upcoming OpenFermion version 1.7.0 release.