bibliothecary
bibliothecary copied to clipboard
librariesio
Improve setup.py parsing
Find dependencies when specified as an array:
required_packages = [
'coverage==3.7.1',
'Mako==1.0.3',
'PyYAML==3.11',
]
Hey there -- since I'm working on the new PyPI and getting it closer and closer to launch, I wanted to check whether you're planning on using the API instead of parsing setup.py -- or perhaps I am misunderstanding your needs?
@brainwane thanks for getting in touch, at the moment we don't want to execute any arbitrary python code so parsing with a regex is the best way for us to handle it until we have a safe way of using each package managers tooling directly, then we'll use the API, thanks!
We are using the pypi rest api in the main Libraries.io codebase over here: https://github.com/librariesio/libraries.io/blob/master/app/models/package_manager/pypi.rb but that doesn't help for this project which parses any setup.py, usually from applications rather than pypi modules.
https://github.com/pypa/twine/blob/fb062b77c7ef38c28430dd4c8edaa592c4b59b07/setup.py#L21:L27 is an example here:
install_requires = [
"tqdm >= 4.14",
"pkginfo >= 1.0",
"requests >= 2.5.0, != 2.15, != 2.16",
"requests-toolbelt >= 0.8.0",
"setuptools >= 0.7.0",
]
@brainwane this library is also used for parsing dependency information from repositories that aren't pushed to pypi, including public and private github repositories, forks etc
Could I please ask for this and the other PyPI issues -- #414, #459, #415 -- to get higher priorities on Tidelift's roadmap? The current gaps in coverage leave a lot of real dependencies untracked.
@brainwane noted!
One thing to be sure is clear to anyone who might stumble on the discussion, these are all for repository dependencies, so https://libraries.io/github/pytest-dev/pytest vs package deps like https://libraries.io/pypi/pytest/5.0.1/tree ... hopefully we're getting the package deps right.
Team Tidelift is very development-team-size-constrained right now, to set expectations, but I do appreciate the bump on these and we've added them to our internal backlog too.
You won't be able to parse setup.py with Ruby. There are no even AST parsers for Python written in Ruby, and no those that can detect syntax for Python 2 and various 3.x versions without errors.
Do you also plan to parse the conditional or extras dependencies in setup.py? Or should I get setup.py to generate the requirements.txt file in order for my dependencies to be accounted for here?
