dffml shouldi: deptree: Create dependency tree of project

https://github.com/intel/dffml/commits/shouldi_dep_tree

The idea behind the work that was done so far in the above branch was to produce the full dependency tree for a given python package. We'll want this to be the shouldi deptree command (as opposed to shouldi install)

It's currently in a state where it can grab package names out of setup.py files. It also needs to be able to grab them out of setup.cfg files and requirements.txt files.

Another thing that's missing is the version number of the package. Right now the latest version of each package is being downloaded. We need to check if the package was pinned to a version and download that version (if example_package==0.3.1 for example we'd download version 0.3.1 of example_package). Come up with reasonable ways to handle all the following cases: https://stackoverflow.com/a/50842265/12310488

There is also a new output operation that needs to be made for this work. It's the Tree output operation. It's not working at the moment. What it should do is output a dict similar to the way config structures work with their use of plugin and config (you'll see what I'm about when you read the existing code.

References
- https://pip.pypa.io/en/stable/reference/requirements-file-format/
- https://github.com/intel/dffml/blob/alice/docs/tutorials/rolling_alice/0001_coach_alice/0001_down_the_dependency_rabbit_hole_again.md
  - https://www.youtube.com/watch?v=D9puJiKKKS8&list=PLtzAOVTpO2jYt71umwc-ze6OmwwCIMnLw&t=871s
- https://github.com/pdxjohnny/use-cases/blob/openssf_metrics/openssf_metrics.md
Example files
- requirements-dev.txt
- service/http/setup.cfg

https://github.com/intel/dffml/blob/0a2e053f5f8e361054f329a3f763982fb1e4d1f7/examples/shouldi/tests/test_dep_tree.py#L1-L169

Apr 29 '20 22:04 johnandersen777

This would probably help us figure out what's wrong with #737

Jun 24 '20 05:06 johnandersen777

Aligned work
- intel/TCF has depenedncies in different langagues
  - SImilar to some our our tokei and the like
  - They created an .nreqs format which is a combinaion of rpm
- https://github.com/chainguard-dev/apko

Oct 19 '22 17:10 johnandersen777

Cross ref to 2022-10-25 Alice Initiative welcome aboard!
- Reproduced bellow
Harsh joining us to do some Python package analysis work

2022-10-25 Alice Initiative welcome aboard!

Harsh joining us to do some Python package analysis work
Alice thread: https://github.com/intel/dffml/discussions/1406?sort=new
This work feeds into the following tutorial
- https://github.com/intel/dffml/blob/alice/docs/tutorials/rolling_alice/0001_coach_alice/0001_down_the_dependency_rabbit_hole_again.md
shouldi: deptree: Create dependency tree of project · Issue #596 · intel/dffml
- https://github.com/intel/dffml/commits/shouldi_dep_tree
- The idea behind the work that was done so far in the above branch was to produce the full dependency tree for a given python package.
Documentation writing process
- https://github.com/intel/dffml/blob/alice/docs/tutorials/rolling_alice/0000_architecting_alice/0004_writing_the_wave.md#vision
Contributing Documentation
- https://github.com/intel/dffml/blob/alice/entities/alice/CONTRIBUTING.rst
Troubleshooting — DFFML fd401e426 documentation
- https://intel.github.io/dffml/main/troubleshooting.html#entrypointnotfound
Next steps
- Harsh will first focus on filling out the other two functions with unit tests for different file contents
  - These functions / files can be standalone at first, we can integrate later.
    - https://github.com/intel/dffml/blob/alice/entities/alice/CONTRIBUTING.rst#writing-an-overlay
  - Harsh to ping John as needed.
  - Harsh to comment in issue with commands run and errors and so forth so we can copy pate into the associated tutorial later.
    - Plans for automation of documentation writing: https://github.com/intel/dffml/commit/74781303fae19b03326878d184a49ac93543749c?short_path=76e9bfe#diff-76e9bfe1c05d4426559fada22595ca1f9a76fd0fc98609dfbbde353d10fa77db

https://github.com/intel/dffml/blob/0a2e053f5f8e361054f329a3f763982fb1e4d1f7/examples/shouldi/tests/test_dep_tree.py#L36-L71

Oct 25 '22 20:10 johnandersen777

2022-11-02 Harsh/John

https://github.com/intel/dffml/discussions/1406?sort=new#discussioncomment-4041436
https://pip.pypa.io/en/stable/reference/requirements-file-format/
Installed VS Code build tools and used the developer prompt from there and it worked
Remembered pipdeptree exists
We should use https://github.com/tox-dev/pipdeptree and integrate that into shouldi.

  -j, --json            Display dependency tree as json. This will yield "raw"
                        output that may be used by external tools. This option
                        overrides all other options.

https://intel.github.io/dffml/main/examples/shouldi.html
https://intel.github.io/dffml/main/contributing/dev_env.html

$ git clone https://github.com/intel/dffml
$ cd dffml
$ python -m venv .venv
$ git checkout -b deptree
$ . .venv/Scripts/activate
$ pip install -e .[dev]
$ cd examples/shouldi
$ pip install -e .[dev]

https://intel.github.io/dffml/main/api/util/packaging.html#dffml.util.packaging.mkvenv
https://github.com/tox-dev/pipdeptree#running-in-virtualenvs

https://github.com/intel/dffml/blob/b892cfab9bd152c47a709e8708491c95b8c3ec8e/tests/docs/test_consoletest.py#L14

Basic testcase will be to analyze shouldi itself

https://github.com/intel/dffml/blob/3530ee0d20d1062605f82d1f5055f455f8c2c68f/dffml/util/testing/consoletest/commands.py#L83-L190

Opens
- Pip not installing to virtualenv we created (using different Python despite our current efforts)
TODO
- [ ] Harsh to investigate refactoring ActivateVirtualEnvCommand into something that doesn't mess with os.environ and behaves more like mkvenv() (https://github.com/intel/dffml/tree/main/dffml/util/testing/consoletest/)
  - [ ] Explicitly use path returned from venv creation as zeroith argument to dffml.run_command()/subprocess.check_call()

Nov 02 '22 20:11 johnandersen777

A discussion today revealed we should go with the file paring approach (rather than pipdeptree)

Nov 10 '22 23:11 johnandersen777

https://github.com/anthonyharrison/sbom4python
- Perfect!!!

$ sbom4python  --format json --sbom spdx -m black                     
{                                                                                                                                                                                         
  "SPDXID": "SPDXRef-DOCUMENT",                                                                                                                                                           
  "spdxVersion": "SPDX-2.2",         
  "creationInfo": {                 
    "comment": "This document has been automatically generated.",                            
    "creators": [                                                                            
      "Tool: sbom4python-0.7.0"                                                              
    ],                                                                                                                                                                                    
    "created": "2023-03-02T13:53:48Z",                                                       
    "licenseListVersion": "3.18"                                                                                                                                                          
  },                                      
  "name": "black",                                                                                                                                                                        
  "dataLicense": "CC0-1.0",                                                                  
  "documentNamespace": "http://spdx.org/spdxdocs/black-7ebf1fb9-a781-41f5-9e57-a6bba6969ecb", 
  "packages": [                                                                                                                                                                           
    {                                                                                        
      "SPDXID": "SPDXRef-Package-1-black",                                                   
      "name": "black",                                                                       
      "versionInfo": "23.1.1.dev8+g25d886f",                                                 
      "supplier": "Organization: ukasz Langa ([email protected])",                             
      "downloadLocation": "NONE",                                                            
      "filesAnalyzed": false,                                                                
      "licenseConcluded": "MIT",                                                             
      "licenseDeclared": "MIT",                                                              
      "copyrightText": "NOASSERTION",  
      "externalRefs": [              
        {                                                                                    
          "referenceCategory": "PACKAGE-MANAGER",                                            
          "referenceLocator": "pkg:pypi/[email protected]+g25d886f",
          "referenceType": "purl"                                                            
        },                       
        {                                                                                                                                                                                 
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:ukasz_langa:black:23.1.1.dev8+g25d886f:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-2-click",
      "name": "click",
      "versionInfo": "8.1.3",
      "supplier": "Organization: Armin Ronacher ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "BSD-3-Clause",
      "licenseDeclared": "BSD-3-Clause",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:pypi/[email protected]",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:armin_ronacher:click:8.1.3:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-3-mypy-extensions",
      "name": "mypy-extensions",
      "versionInfo": "1.0.0",
      "supplier": "Organization: The mypy developers ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "MIT",
      "licenseDeclared": "MIT",
      "copyrightText": "NOASSERTION",
      "externalRefs": [                                                                                                                                                           [97/296]
        {                                                                                    
          "referenceCategory": "PACKAGE-MANAGER",                                            
          "referenceLocator": "pkg:pypi/[email protected]+g25d886f",
          "referenceType": "purl"                                                            
        },                       
        {                                                                                                                                                                                 
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:ukasz_langa:black:23.1.1.dev8+g25d886f:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-2-click",
      "name": "click",
      "versionInfo": "8.1.3",
      "supplier": "Organization: Armin Ronacher ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "BSD-3-Clause",
      "licenseDeclared": "BSD-3-Clause",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:pypi/[email protected]",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:armin_ronacher:click:8.1.3:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-3-mypy-extensions",
      "name": "mypy-extensions",
      "versionInfo": "1.0.0",
      "supplier": "Organization: The mypy developers ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "MIT",
      "licenseDeclared": "MIT",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:pypi/[email protected]",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:the_mypy_developers:mypy-extensions:1.0.0:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-4-packaging", 
      "name": "packaging",
      "versionInfo": "23.0",
      "supplier": "Organization: Donald Stufft ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "NOASSERTION",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:pypi/[email protected]",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:donald_stufft:packaging:23.0:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-5-pathspec",
      "name": "pathspec",
      "versionInfo": "0.11.0",
      "supplier": "Organization: Caleb P. ([email protected])",
      "downloadLocation": "NONE",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "NOASSERTION",
      "copyrightText": "NOASSERTION",
      "externalRefs": [

Mar 02 '23 13:03 johnandersen777