Problem with Installation/Config

[ccosborne]

Hi there,

I am running into problems during installation when I execute the python3 setup.py develop --user command. For context, my previous steps are: (1) clone bigbang repo to remote server, (2) create virtualenv, and (3) execute pip3 install -r requirements.txt. When I run python3 setup.py develop --user, I get the output pasted below. After this, I add the virtualenv as a jupyter kernel, open an example notebook (Git Collection), and when I try to import the packages I get: ModuleNotFoundError: No module named 'config'. I assume this is linked to the problems with executing setup.py.

Do you know how to troubleshoot this? Thanks!

(venv-bigbang) user1@server1:~/bigbang$ python3 setup.py develop --user
/home/user1/venv-bigbang/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'dependencies' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `dependencies = ['beautifulsoup4>=4.9.3', 'chardet>=4.0.0', 'common>=0.1.2', 'enlighten>=1.7.2', 'gender-detector', 'GitPython', 'html2text>=2020.1.16', 'ietfdata>=0.4.0', 'ipython>=7.22.0', 'jinja2>=2.11.3', 'jupyter>=1.0.0', 'jsonschema>=3.2.0', 'matplotlib>=3.4.1', 'networkx>=2.5.1', 'nltk>=3.6.2', 'numpy>=1.20.2', 'pandas>=1.2.3', 'powerlaw>=1.4.6', 'python-dateutil>=2.8.1', 'python-Levenshtein>=0.12.2', 'pytz>=2021.1', 'pyyaml>=5.4.1', 'pyzmq>=22.0.3', 'requests>=2.25.1', 'seaborn', 'tornado>=6.1', 'tqdm>=4.60.0', 'validator_collection>=1.5.0', 'python-docx>=0.8']`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'dependencies' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'dependencies' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/home/user1/venv-bigbang/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'optional-dependencies' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `optional-dependencies = {'test': ['pytest>=6.2.3', 'coverage>=5.5', 'testfixtures>=6.17.1'], 'dev': ['pytest>=6.2.3', 'coverage>=5.5', 'testfixtures>=6.17.1', 'black>=20.8b1', 'isort>=5.7.0', 'pre-commit>=2.13.0']}`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'optional-dependencies' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'optional-dependencies' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
running develop
/home/user1/venv-bigbang/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
WARNING: The user site-packages directory is disabled.
/home/user1/venv-bigbang/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Checking .pth file support in /home/user1/.local/lib/python3.8/site-packages
/home/user1/venv-bigbang/bin/python3 -E -c pass
TEST FAILED: /home/user1/.local/lib/python3.8/site-packages does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/user1/.local/lib/python3.8/site-packages

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.
pre-commit installed at .git/hooks/pre-commit

[sbenthall]

Yes, thanks for sending along the installation errors. It sheds light on this.

It looks like there are two separate issues. One has to do with the error in pyproject.toml.

But what is more likely causing the error when importing config is the problem with the python path.

Googling for this error reveals that the problem might be solved by removing the option --user from the installation command:

https://github.com/NVIDIA/flownet2-pytorch/issues/183

This may be a problem with our installation instructions, as --user can conflict with virtualenv functionality.

[ccosborne]

Thanks for your help, Seb.

Removing --user removed the PYTHONPATH error but the pyproject.toml error persists. The error message says I should list dependencies and optional-dependencies under dynamic - do you know how to do that?

[sbenthall]

@ccosborne The issues with pyproject.toml appear to be warnings, not errors. They are about parts of the build system that are soon to be (but not yet) deprecated.

Thanks for bringing that to our attention!

But it should not be blocking the execution of your Jupyter notebook.

Do you still have problems with import config?

[ccosborne]

@sbenthall Unfortunately I get the same error message with import config in the Jupyter notebook. Any clue why that might be?

[sbenthall]

Hello @ccosborne I see the issue. It's a problem with the configuration on our end. I'll work on this and let you know when I've pushed the change.

[sbenthall]

Ok, I've pushed a change and tests are passing. Can you please try the installation again?

[ccosborne]

Unfortunately I get the same warning message when running python3 setup.py develop and the same error message when importing from bigbang.archive import Archive in the notebook, even after deleting bigbang and cloning it again... :/

[MridulS]

This is indeed an issue, and this pops up because the config module isn't properly packaged. It's a sibling to the bigbang module and there isn't any way to automatically make config importable unless explicitly adding it the the python system paths.

The tests will pass because they are running in the root dir so they can do import config without ever having this installed properly. This can be fixed with some adding the paths to sys.paths but I would highly discourage that. The best way here would be to restructure the package along the lines of:

- bigbang/
     - bigbang
     - config
     - tests
     - bin

with from config.config import CONFIG becoming something like from bigbang.config import CONFIG.

Let me know if I am missing something here regarding the structure of bigbang (and thus the current structure).

[sbenthall]

@MridulS Thanks so much for this analysis!

Not a lot of thought was put into the location of the config file. I will implement your suggestions here.

One question though -- in another project I've been able to import sibling modules by adding them to the packages list: https://github.com/sbenthall/SHARKFin/blob/master/setup.cfg#L18 Is there something wrong with that approach?

Ultimately, we use the config module just to keep track of a bunch of default file system paths and regular expressions. We want to access them from Python, but store them in a YAML file.

Having them bundled with the bigbang source code means that there would be no way for a 'user' who is just using BigBang in development mode to modify that file -- they would need to look around in their site-packages or something.

I wonder @MridulS how you would approach this from the perspective of the on-line dashboard, which I believe is built on the package available on PyPI and not a development installation. Is there a better way to expose these configuration options?

[MridulS]

One question though -- in another project I've been able to import sibling modules by adding them to the packages list: https://github.com/sbenthall/SHARKFin/blob/master/setup.cfg#L18 Is there something wrong with that approach?

This does indeed work! But the issue here is that sharkfin, simulate and pnl_market are essentially different "packages". They will work just fine in isolation but as soon as someone wants to use them as an external library it can mess things up, like simulate is way common of a name and there is no way a person can import import sharkfin.simulate. When I do pip install sharkfin it shouldn't install simulate or pnl_market IMO (this is one of my annoyance with econ-ark too, the distribution name should match the actual package) This looks very close to namespace packages. I would rather just have the standard layout unless there is some proper need for this.

Ultimately, we use the config module just to keep track of a bunch of default file system paths and regular expressions. We want to access them from Python, but store them in a YAML file.

Having them bundled with the bigbang source code means that there would be no way for a 'user' who is just using BigBang in development mode to modify that file -- they would need to look around in their site-packages or something.

This can live under the bigbang folder too and I think we can make the whole config bits better with using Pathlib, it would lead to much cleaner code. The user can write their own YAML file and the initial load of bigbang can read that in (the user can provide an explicit path).

I wonder @MridulS how you would approach this from the perspective of the on-line dashboard, which I believe is built on the package available on PyPI and not a development installation. Is there a better way to expose these configuration options?

Currently bigbang isn't usuable to build a dashboard or as an external library, (atleast while using the CLBE tutorial). The pypi version breaks (atleast the latest one at https://pypi.org/project/bigbang-py/) when I try to import Archive. In a fresh env:

>>> from bigbang.archive import Archive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ms/mambaforge/envs/env-test-0/lib/python3.10/site-packages/bigbang/__init__.py", line 1, in <module>
    from . import ingress
ImportError: cannot import name 'ingress' from partially initialized module 'bigbang' (most likely due to a circular import) (/Users/ms/mambaforge/envs/env-test-0/lib/python3.10/site-packages/bigbang/__init__.py)

We need to change the structure and make sure the imports are fixed :)

To get around for now I have moved around the files and fixed some imports locally. I can send the changes as a PR but it's pretty hackish and the config folder should probably change anyway.

[sbenthall]

Thanks so much @MridulS

This is all my fault -- I packaged BigBang for PyPI not long ago, but clearly made a mess of it.

Any PRs you make along these lines would be greatly appreciated. I defer to your guidance on these things.