PyPI query used for ingestion no longer contains releases info (breaking pypi-filter)

[MagielBruntink]

The REST API uses the following URL in the lazy ingestion code, for example for namesss/0.1.0: https://pypi.org/pypi/namesss/0.1.0/json

However, the JSON returned by the PyPI API is now too limited; it no longer contains releases information. The PyPI Filter then logs: Could not retrieve packaging info

[MagielBruntink]

This query does return the releases info: https://pypi.org/pypi/namesss/json I'm not sure what is broken actually, the PyPI API, the ingestion code, or the pypy-filter plugin.

[gdrosos]

Thanks @MagielBruntink for reporting the issue!

Indeed, the PyPI API endpoint:/pypi/<project_name>/<project_version>json previously would return a json field releases with information regarding all the releases of a specific package. This information was used by the PyPI-Filter in order to ingest all the non-existing PyPI versions. Unfortunately, according to the PYPI Warehouse Documentation: "Previously this response included the releases key, which had the URLs for all files for every release of this project on PyPI. Due to stability concerns, this had to be removed from the release specific page, which now only serves data specific to that release. To access all files, you should preferrably use the simple API, or otherwise use the non versioned json api at /pypi/<project_name>/json."

Therefore, we could ingest the missing package versions from the /pypi/<project_name>/json entrypoints in the REST-API & PyPI-Filter. I could work on this starting from Friday.

[MagielBruntink]

Hi @gdrosos are you able to progress on this issue?