feat(newAgent): compatibility agent

[GMishx]

Description

New compatibility agent for FOSSology which can perform following tasks:

1. Compare licenses found in a single file and highlight incompatible licenses in tree view.
2. Compare licenses found in the upload and main license of the upload for auto decision of licenses.
3. Custom license compatibility list for entire server. Can be exported/imported as YAML file and be used standalone in CLI mode.
4. Import license rules from OSADL compatibility matrix.

New field License Type

This PR also adds a new field for licenses called as license type which by default has 3 possible values "Permissive", "Weak Copyleft" and "Strong Copyleft". The list can be altered from the UI in customize section. Each license can then be associated with 1 type.

This allows simplification of license rules as licenses can be clubbed together as a group and compared with other in following possible ways:

1. License A with License B (direct comparison between 2 licenses)
2. License A with License Type B (comparison of a license with whole group of licenses)
3. License Type A with License Type B (comparison between 2 license groups)

This greatly reduces the complexity of license rules on the expense of assigning of each license to a group.

Importing license rules from OSADL matrix

Once all licenses in the server has been assigned a license type, the script utils/osadl_convertor.py. To run the script, install the required dependencies using following pip install:

python3 -m pip install -r utils/requirements.osadl.txt

Then run the script with following parameters. It will create a YAML file which contains all the rules from OSADL compatibility matrix and reduce the rule list size as much as possible with help of license grouping (for which it needs read access to DB).

python3 utils/osadl_convertor.py [--user USER] --password PASSWORD [--database DATABASE] [--host HOST] [--port PORT] --yaml YAML [-d]

Auto decision of licenses

While performing an upload and scan, in the section 8 "Automatic Concluded License Decider", the option "Auto conclude license finding if they are of type" can be selected with a license type. If a file contain only licenses of given type, and contains no rules which disallow them to be used together, will be marked with identified decision.

The same can be done with REST API and even provided with a main license for the upload.

curl -X 'POST' \
  'http://localhost/repo/api/v1/uploads/2/licenses/main' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer token' \
  -H 'Content-Type: application/json' \
  -d '{
  "shortName": "MIT"
}'

Then run the decider

curl -X 'POST' \
  'http://localhost/repo/api/v1/jobs' \
  -H 'accept: application/json' \
  -H 'folderId: 2' \
  -H 'uploadId: 2' \
  -H 'Authorization: Bearer token' \
  -H 'Content-Type: application/json' \
  -d '{
  "analysis": {
    "compatibility": true
  },
  "decider": {
    "conclude_license_type": "Permissive"
  }
}'

Above 2 requests will set the main license of upload 2 as "MIT" and auto decide all files without compatibility issues where license type is "Permissive".

Changes

1. New agent compatibility.
2. 2 new tables license_rules to hold the rules and comp_result to hold compatibility results.
3. Refactoring some supported files and test cases.

How to test

1. Install the branch and create new license types from Admin > Customize.
2. Associate licenses to different license groups.
3. Create custom rules from Admin > License Admin > Compatibility Rules.
4. Upload test package and scan. At the same time schedule the compatibility agent.
5. Files where license rules do not allow licenses to be used together should be highlighted with bold RED color.
6. Run the decider agent with a license type. All correct files should get auto decided.
7. Run the OSADL script and import the resulting YAML file from Admin > License Admin > Rules Import.
8. Check the resultant rules.

[github-actions[bot]]

This pull request has conflicts, please rebase with master to resolve those before we can evaluate the pull request.

[shaheemazmalmmd]

Error While executing osadl_convertor.py File "osadl_convertor.py", line 2

SyntaxError: Non-ASCII character '\xc2' in file osadl_convertor.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details.

[shaheemazmalmmd]

Please fix the warning. and rebase with latest master.

Wed Sep 25 14:14:24.544047 2024] [php:warn] [pid 98532] [client 127.0.0.1:41498] PHP Warning: Trying to access array offset on false in /usr/local/share/fossology/lib/php/Application/LicenseCompatibilityRulesYamlImport.php on line 202, referer: http://localhost/repo/?mod=admin_license_from_yaml

[GMishx]

Please fix the warning. and rebase with latest master.

Wed Sep 25 14:14:24.544047 2024] [php:warn] [pid 98532] [client 127.0.0.1:41498] PHP Warning: Trying to access array offset on false in /usr/local/share/fossology/lib/php/Application/LicenseCompatibilityRulesYamlImport.php on line 202, referer: http://localhost/repo/?mod=admin_license_from_yaml

Resolved. Please test again.