SourceRank check does not detect README.rst

[schettino72]

Bugs

It is common for python projects to name their README file with a rst extension. Although my project [1] contains a README.rst, SourceRank shows my project as not having a README file.

https://libraries.io/pypi/doit/sourcerank

[movermeyer]

I'm not a maintainer but I think I can answer this

---

libraries.io can handle README.rst (ex. simplejson). That's not the problem.

The issue is that libraries.io is not able to detect the source repo from the PyPI metadata. You can see this in doit's SourceRank, where Source repository present? is set to 0. If it could find the GitHub repo, then it would be able to find the README.rst.

If it could find it, doit would get at least +2 to its SourceRank (+1 for knowing the repo, +1 for knowing the README), plus some for its GitHub stars.

---

Currently libraries.io only looks at the ['info']['home_page'] field of the PyPI API's JSON info about doit.

I see that doit lists their code repo in "project_urls" in setup.py. That seems reasonable, but it looks like PyPI doesn't expose that information in their API, so libraries.io can't access it automatically.

In the future, if PyPI were to expose that info, it seems like it would be easy to add a few heuristics to pull out that value and use it as the repository_url value.

Perhaps the next step would be to get PyPI to expose this information?

[movermeyer]

It seems that PyPI has already been working on this and is actually about to merge the changes needed.

So this information could be exposed any day now!

[schettino72]

Hi @movermeyer,

Thanks for looking into this.

Are you saying libraries.io will look for source in GIT repository but not in the files distributed in a release? :open_mouth:

You can see this in doit's SourceRank, where Source repository present? is set to 0.

Actually I changed setup.py to include project_urls after latest doit release. I guess that's why libraries.io still does not know about it...

I was not aware of this configuration, I found exactly because I also wanted the star counted to be displayed on my project page. :smile:

[movermeyer]

Are you saying libraries.io will look for source in GIT repository but not in the files distributed in a release? :open_mouth:

Yes. If this limitation is intentional, I imagine it is done to keep things simple.

README's are often not included in releases, especially binary distributions. For example, the wheel version of doit doesn't include the README. (I know that in doit's case there is also a source distribution that includes the README, but that's not true for many libraries).

Currently, much of libraries.io is pretty simple. There's not a whole lot of fancy heuristics or deep analysis going on. That said, doing that sort of analysis might be a good future step.

[Jehan]

Hi!

I notice that the 'project_urls' dict value is now merged in since July: https://github.com/pypa/warehouse/pull/4184

And this is also in Python docs: https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls

So if we add the project_urls['Source'] value to a git repository (not necessarily on github), it would be nice if libraries.io could now set +1 for the repo, and also +1 if it detects the README.rst.

[peterjc]

This appears to have been fixed - doit has both the repository and readme ticked: https://libraries.io/pypi/doit/sourcerank

[schettino72]

This appears to have been fixed - doit has both the repository and readme ticked: https://libraries.io/pypi/doit/sourcerank

I (doit) learned how to add repository info to setup.py. So that does not mean the issue was fixed.

[peterjc]

So you consider the bug to be the README check only works if the repository link was found? i.e. It should check the PyPI source code releases or something?

[schettino72]

IMO bug is that it didnt work as I expected. If you dont want to support reading setup.py, at least document it clearly... Not sure what is the situation now but in order to figure out what to do I had to check a project where "README" was detected. From that I understood how to include repository information.

So before saying README is not present you should say "repository information" not present!

[peterjc]

I agree, having multiple tests contingent on finding a repository seems flawed.

On the bright side, I'm now aware of the "project_urls" setting in setup.py and will start using it.

[vivi90]

My problem I still have the same issues with my package vmcp:

• Source repository present?: 0
• Readme present?: 0

What i have My pyproject.toml:

[build-system]
requires = ["setuptools>=61.0", "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "vmcp"
authors = [{name="Vivien Richter", email="[email protected]" }]
description = "Virtual Motion Capture protocol package"
keywords = ["VMC", "protocol", "MR", "VR", "XR", "AR"]
readme = "README.md"
license = {text="AGPL-3.0-or-later"}
requires-python = ">=3.10"
classifiers = [
    "Programming Language :: Python :: 3",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Topic :: Internet",
    "Topic :: Multimedia :: Graphics :: Capture",
    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
    "Operating System :: OS Independent",
    "Development Status :: 5 - Production/Stable"
]
dynamic = ["version"]

[project.optional-dependencies]
osc4py3 = ["osc4py3 >= 1.0.1, < 1.0.4"]
docu = [
    "pdoc",
    "setuptools_scm"
]
tests = [
    "pylint",
    "mypy",
    "pytest",
    "bandit"
]
dev = [
    "pylint",
    "mypy",
    "pytest",
    "bandit",
    "black",
    "flake8",
    "pydocstyle"
]

[project.urls]
homepage = "https://codeberg.org/vivi90/python-vmcp"
documentation = "https://vivi90.codeberg.page/python-vmcp"
repository = "https://codeberg.org/vivi90/python-vmcp"
tracker = "https://codeberg.org/vivi90/python-vmcp/issues"

# Dynamic versioning
[tool.setuptools_scm]

[tool.mypy]
python_version = "3.10"

# Suppress missing imports, due to missing library stubs or py.typed marker
[[tool.mypy.overrides]]
module = [
    "setuptools",
    "pytest.*",
    "osc4py3.*"
]
ignore_missing_imports = true

[tool.pytest.ini_options]
addopts = "--doctest-modules"
testpaths = "tests"

According to PEP 621 this should work.

Maybe helpful In contrast the package pyproject-toml works. Even if they have the same pyproject.toml structure like me:

[build-system]
requires = ["pyproject-toml>=0.0.10"]
build-backend = "pyproject_toml.build_system.build_backend"
backend-path = ["."]

[project]
name = "pyproject-toml"
version = "0.0.11"
description = "Project intend to implement PEP 517, 518, 621, 631 and so on."
readme = "README.rst"
authors = [{ name = "Tang Ziya", email = "[email protected]" }]
license = { text = "MIT" }
classifiers = [
    "Development Status :: 2 - Pre-Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
]
dependencies = [
    "setuptools >= 42",
    "wheel",
    "toml",
    "jsonschema",
    "typing-extensions; python_version < '3.8'",
]

[project.urls]
homepage = "https://github.com/pyproject-toml/pyproject-toml"

[tool.pyproject-toml]
packages = ["pyproject_toml", "pyproject_toml.build_system", "pyproject_toml.tool"]

[tool.tox]
legacy_tox_ini = """
[tox]
envlist = py{36,37,38,39,310}
isolated_build = True
[testenv]
deps =
    jaraco.path
    pytest
    pytest-cov
    setuptools
commands = pytest
"""

I don't see the difference 🤔

[Wikilicious]

I have spent too much time trying to figure this out... Using pyproject.toml project.urls... PYPI is able to figure out source code, issue tracker, README, and documentation URLs but libraries.io fails.

[YisusChrist]

My problem I still have the same issues with my package vmcp:

• Source repository present?: 0
• Readme present?: 0

What i have My pyproject.toml:

[build-system]
requires = ["setuptools>=61.0", "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "vmcp"
authors = [{name="Vivien Richter", email="[email protected]" }]
description = "Virtual Motion Capture protocol package"
keywords = ["VMC", "protocol", "MR", "VR", "XR", "AR"]
readme = "README.md"
license = {text="AGPL-3.0-or-later"}
requires-python = ">=3.10"
classifiers = [
    "Programming Language :: Python :: 3",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Topic :: Internet",
    "Topic :: Multimedia :: Graphics :: Capture",
    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
    "Operating System :: OS Independent",
    "Development Status :: 5 - Production/Stable"
]
dynamic = ["version"]

[project.optional-dependencies]
osc4py3 = ["osc4py3 >= 1.0.1, < 1.0.4"]
docu = [
    "pdoc",
    "setuptools_scm"
]
tests = [
    "pylint",
    "mypy",
    "pytest",
    "bandit"
]
dev = [
    "pylint",
    "mypy",
    "pytest",
    "bandit",
    "black",
    "flake8",
    "pydocstyle"
]

[project.urls]
homepage = "https://codeberg.org/vivi90/python-vmcp"
documentation = "https://vivi90.codeberg.page/python-vmcp"
repository = "https://codeberg.org/vivi90/python-vmcp"
tracker = "https://codeberg.org/vivi90/python-vmcp/issues"

# Dynamic versioning
[tool.setuptools_scm]

[tool.mypy]
python_version = "3.10"

# Suppress missing imports, due to missing library stubs or py.typed marker
[[tool.mypy.overrides]]
module = [
    "setuptools",
    "pytest.*",
    "osc4py3.*"
]
ignore_missing_imports = true

[tool.pytest.ini_options]
addopts = "--doctest-modules"
testpaths = "tests"

According to PEP 621 this should work.

Maybe helpful In contrast the package pyproject-toml works. Even if they have the same pyproject.toml structure like me:

[build-system]
requires = ["pyproject-toml>=0.0.10"]
build-backend = "pyproject_toml.build_system.build_backend"
backend-path = ["."]

[project]
name = "pyproject-toml"
version = "0.0.11"
description = "Project intend to implement PEP 517, 518, 621, 631 and so on."
readme = "README.rst"
authors = [{ name = "Tang Ziya", email = "[email protected]" }]
license = { text = "MIT" }
classifiers = [
    "Development Status :: 2 - Pre-Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
]
dependencies = [
    "setuptools >= 42",
    "wheel",
    "toml",
    "jsonschema",
    "typing-extensions; python_version < '3.8'",
]

[project.urls]
homepage = "https://github.com/pyproject-toml/pyproject-toml"

[tool.pyproject-toml]
packages = ["pyproject_toml", "pyproject_toml.build_system", "pyproject_toml.tool"]

[tool.tox]
legacy_tox_ini = """
[tox]
envlist = py{36,37,38,39,310}
isolated_build = True
[testenv]
deps =
    jaraco.path
    pytest
    pytest-cov
    setuptools
commands = pytest
"""

I don't see the difference 🤔

Same happens to me, it just doesn't make sense. It may be something related to the package metadata but I'm not sure