REST APIs Endpoints for licensing.

[michelescarlato]

Hi @mir-am , @MagielBruntink and @gdrosos ,

I tried to unify as much as I could in the metadata augmented field with license information, in order to simplify the job of having licensing endpoints for each ecosystem (Maven, PyPI, and Debian).

This is the current status:

1. Debian:

• package_versions table:

277 |        277 | 1.0.0-2                              | cscout       |                        | amd64        | 2017-12-25 16:13:42 | {"licenses": [{"name": "GPL-2+", "source": "DEBIAN_PACKAGES"}]}

• files table (with the updated version, which still has not been re-deployed on Monster):

15 |                  1 | compat/strtoi.c                                    |          |            | {"licenses": [{"name": "BSD-2", "source": "DEBIAN_PACKAGES"}]}

• files table (not updated version, and currently on Monster):

318189 |                442 | apps/alsamixer/mixer.c                                                                        |          |            | {"license": "GPL-3+"}

2. PyPI:

• package_versions table:

84883 |      28519 | 0.35.0                                                              | PyCG         |                        |              | 2019-06-19 17:10:16 | {"licenses": [{"name": "Apache-2.0", "source": "GITHUB"}]}

• files table:

20464061 |             637072 | providers/salesforce/hooks/__init__.py                                                                                                                                                                                         |          |            | {"licenses": [{"key": "apache-2.0", "spdx_license_key": "Apache-2.0"}, {"key": "apache-2.0", "spdx_license_key": "Apache-2.0"}]}

3. Maven:

• package_versions table:

169282 |      22672 | 1.0-rc1                                                        | OPAL         |                     -1 |              | 2014-01-22 18:15:52 | {"forge": "mvn", "groupId": "com.google.auto.service", "repoUrl": "scm:git:git://github.com/google/auto.git", "version": "1.0-rc1", "licenses": [{"name": "Apache-2.0", "source": "GITHUB"}], "commitTag": "HEAD", "artifactId": "auto-service", "sourcesUrl": "https://repo.maven.apache.org/maven2/com/google/auto/service/auto-service/1.0-rc1/auto-service-1.0-rc1-sources.jar", "projectName": "AutoService", "releaseDate": 1390414552000, "dependencies": [{"id": 0, "type": "jar", "scope": "compile", "groupId": "com.google.guava", "optional": false, "artifactId": "guava", "classifier": "", "exclusions": [], "versionConstraints": ["16.0"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "com.google.testing.compile", "optional": false, "artifactId": "compile-testing", "classifier": "", "exclusions": [], "versionConstraints": ["0.3"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "junit", "optional": false, "artifactId": "junit", "classifier": "", "exclusions": [], "versionConstraints": ["4.11"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "org.truth0", "optional": false, "artifactId": "truth", "classifier": "", "exclusions": [], "versionConstraints": ["0.13"]}], "packagingType": "jar", "parentCoordinate": "com.google.auto:auto-parent:pom:1.0-rc1", "artifactRepository": "https://repo.maven.apache.org/maven2/", "dependencyManagement": [{"id": 0, "type": "jar", "scope": "provided", "groupId": "javax.inject", "optional": false, "artifactId": "javax.inject", "classifier": "", "exclusions": [], "versionConstraints": ["1"]}, {"id": 0, "type": "jar", "scope": "compile", "groupId": "com.squareup.dagger", "optional": false, "artifactId": "dagger", "classifier": "", "exclusions": [], "versionConstraints": ["1.2.0"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "org.truth0", "optional": false, "artifactId": "truth", "classifier": "", "exclusions": [], "versionConstraints": ["0.13"]}, {"id": 0, "type": "jar", "scope": "compile", "groupId": "com.squareup.dagger", "optional": true, "artifactId": "dagger-compiler", "classifier": "", "exclusions": [], "versionConstraints": ["1.2.0"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "com.google.testing.compile", "optional": false, "artifactId": "compile-testing", "classifier": "", "exclusions": [], "versionConstraints": ["0.3"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "junit", "optional": false, "artifactId": "junit", "classifier": "", "exclusions": [], "versionConstraints": ["4.11"]}, {"id": 0, "type": "jar", "scope": "compile", "groupId": "com.google.guava", "optional": false, "artifactId": "guava", "classifier": "", "exclusions": [], "versionConstraints": ["16.0"]}, {"id": 0, "type": "jar", "scope": "provided", "groupId": "com.google.code.findbugs", "optional": false, "artifactId": "jsr305", "classifier": "", "exclusions": [], "versionConstraints": ["1.3.9"]}, {"id": 0, "type": "jar", "scope": "compile", "groupId": "com.squareup", "optional": false, "artifactId": "javawriter", "classifier": "", "exclusions": [], "versionConstraints": ["2.4.0"]}, {"id": 0, "type": "jar", "scope": "test", "groupId": "com.google.guava", "optional": false, "artifactId": "guava-testlib", "classifier": "", "exclusions": [], "versionConstraints": ["15.0"]}]}

• files table:

78377 |                439 | com/beust/jcommander/IParameterValidator2.java                                                                                                                                                                                                                                                                        |          |            | {"licenses": [{"key": "apache-2.0", "spdx_license_key": "Apache-2.0"}, {"key": "apache-2.0", "spdx_license_key": "Apache-2.0"}]}

[MagielBruntink]

What use case would a separate endpoint for licensing accomplish? For vulnerabilities we created the endpoints because we maintain a table separate from the metadata fields, which maps vulnerabilities to purls even if package-versions have not been ingested. I don't see need for licenses at the moment. Why not stick with inserting into just the metadata fields?

[michelescarlato]

Well, to perform license compliance verification (which is done through another service, called LCV) we need to get the data from the metadata field and parse them. Isolating the content of licenses : {} would make it clearer.

[MagielBruntink]

Could you not just do GET /mvn/packages/{pkg}/{pkg_ver}/metadata and then grab the licenses field from that?

[michelescarlato]

That is what I am currently doing.

Are the APIs still down?

I am mocking many things while writing the PyPI plugin code.

[MagielBruntink]

The mvn api is down, pypi and Debian are up.

[mir-am]

The Java REST API is down. I am still investigating the issue to solve it.

[michelescarlato]

Sorry guys (@MagielBruntink @mir-am @gdrosos ), If a package/packageVersion is not available on FASTEN, is it correct to receive a 404?

I thought we should provide a 500 error for missing information.

Here there are several calls that return 404:

Receive metadata from FASTEN:
https://api.fasten-project.eu/api/pypi/packages/docopt/0.6.2/metadata
Querying docopt:0.6.2: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/idna/3.3/metadata
Querying idna:3.3: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/urllib3/1.26.9/metadata
Querying urllib3:1.26.9: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/charset-normalizer/2.0.12/metadata
Querying charset-normalizer:2.0.12: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/wheel/0.23.0/metadata
Querying wheel:0.23.0: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/requests/2.28.0/metadata
Querying requests:2.28.0: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/yarg/0.1.9/metadata
Querying yarg:0.1.9: metadata something went wrong.
404
https://api.fasten-project.eu/api/pypi/packages/certifi/2022.5.18.1/metadata
Querying certifi:2022.5.18.1: metadata something went wrong.
404

All of them provide 404 and the message is: Package version not found

EDIT: sorry I just saw here that 404 is for missing packages. We were doing wrong in pypi-plugin.