Create ABOUT files for AboutCode Toolkit

[ThorstenHarter]

Hello, first of all I want to thank the developers for their work, these tools are very helpful.

I have created this issue, because I have a question which might also be of interest for other users.

Reading the descriptions of the AboutCode projects on www.aboutcode.org, I got the impression that the workflow would be like this: source -> ScanCode Toolkit -> AboutCode Manager -> AboutCode Toolkit -> attribution notice

Then I realized, that AboutCode Manager and Toolkit don't collaborate, as the Manager cannot generate ABOUT-files, only the components list as JSON.

Is this assumption correct? If so, would you consider adding this feature?

Best regards, Thorsten Harter

[steven-esser]

Hey @ThorstenHarter, thanks for the ticket.

Generating ABOUT files directly from AboutCode Manager is certainly on the radar as a feature down the road. We can use this ticket to tract that progress.

In the mean time, you can generate ABOUT files for Components created in AboutCode Manager, you just have to use https://github.com/nexB/aboutcode-toolkit for now. This workflow would be:

• Export Components from AboutCode Manager in either JSON or CSV formats
• using AboutCode-Toolkit, run: about gen ~/path/to/exported/components.csv/json ~/path/to/aboutfile/output/

Once you have these ABOUT files created, you can then generate comprehensive attribution notices as needed using the AboutCode workflow.

Some relevant AboutCode-Toolkit docs:

• https://github.com/nexB/aboutcode-toolkit/blob/develop/README.rst
• https://github.com/nexB/aboutcode-toolkit/blob/develop/USAGE.rst

[ThorstenHarter]

Hi Steven, thank you for the quick response.

I tried to generate ABOUT files from the CSV/JSON export like you described, but ran into the following problem:

Running aboutcode-toolkit version 3.0.0
Generating .ABOUT files...
Generated 0 .ABOUT files with 1 errors or warnings
CRITICAL: The essential field 'about_file_path' is not found.

The following fields are exported by AboutCode Manager 2.4 for each component in the CSV file: Status,"Path","Name","Version","Owner","License","Copyright","Modified","Deployed","Code Type","Programming Language","Homepage URL","Download URL","License URL","Notice URL","Feature","Purpose","Notes"

I then tried to activate the --mapping flag and placed a MAPPING.CONFIG with "about_file_path: Path" in the current working directory and in the input directory (not sure where about-code looks for it, would be more clear if you could specifiy the path), but with the same result.

At last I replaced "Path" manually with "about_file_path", which didn't help either.

The AboutCode Toolkit documentation for the gen command says:

Given an inventory of ABOUT files at location, generate ABOUT files in base directory.

This sounds as if the purpose of the command is to aggregate already existing ABOUT files.

Example for an exported component description: Components.json.txt

[steven-esser]

@ThorstenHarter:

The default location where AboutCode-Toolkit looks for the mapping.config file (assuming you cloned/downloaded AboutCode-Toolkit from github.) is in:

src/attributecode/mapping.config

You can modify this file or replace it with your own created mapping.config.

This is certainly not documented well, and I will open a ticket in AboutCode-Toolkit for this specifically (https://github.com/nexB/aboutcode-toolkit/issues/317).

For your specific example:

• modify/replace the mapping.config file in the aboutcode-toolkit/src/attributecode/ directory, with atleast the line: about_file_path: Path
• run the command ./about gen --mapping ~/path/to/components.csv ~/path/to/aboutfile-output-dir/

The ABOUT file(s) should then be successfully generated. Also note that you should export Components from AboutCode-Manager in CSV format for now.

While I stated previously that JSON files can be used by AboutCode-Toolkit, I do not think AboutCode-Manager's custom JSON component export is in the correct format to be readily accepted by AboutCode-Toolkit. I will confirm this an open up issues in the correct projects.

Hopefully this is a bit more clear now. Give it a try and let me know if all goes well! We appreciate getting the feedback; every bit helps us improve our AboutCode tools.

[ThorstenHarter]

@majurg

Editing the mapping.config at the specified location solved the problem of the missing 'about_file_path' field, thanks for the tip!

The gen command generates now the ABOUT-files.

There was one warning though: WARNING: Field modified is present but empty The "Modified" column in the CSV is set to "true" or "false" for each component, so I don't know how this warning could be silenced.

Creating the attribution notice from these ABOUT files worked for the simple cases with only one license and copyright. In cases where I added multiple licenses and copyrights in AboutCode Manager however, these multiple entries are merged to one string in the CSV export, so that the AboutCode Toolkit cannot separate them. Example: license_name: MPL-2.0MINPACKBSD-3-Clause

Having multiple licenses and a long list of copyright holders is a common use case, e.g. in the Eigen library: https://github.com/libigl/eigen

I'm not sure how this could be handled if the AboutCode Manager would export ABOUT files. On the one hand, there is the 'license_expression' field, which let's you concatenate licenses with AND/OR, but on the other hand you can only specifiy a single 'license_file' or 'license_url'.

For the copyrights, it would be nice if each entry appears on the separate line in the attribution notice, this can be done with the following syntax in the ABOUT file:

copyright: |
  Copyright (c) 2017 Alice
  Copyright (c) 2018 Bob

[OmkarPh]

I've created an ABOUT file in a recent PR - https://github.com/nexB/scancode-workbench/blob/fix/packaging-structure/workbench.ABOUT

CC @pombredanne @AyanSinhaMahapatra