Improving Score Reporting: Empty repository

[azeemshaikh38]

Running Scorecard on an empty repository (no commits) returns results which do not make sense:

./scorecard --repo=azeemshaikh38/empty-repo-test --format=json | jq .
2022/08/17 01:00:47 unable to get tarball tarball not found: https://api.github.com/repos/azeemshaikh38/empty-repo-test/tarball/. Skipping...                                                        
{                                                                                                                                                                                                    
  "date": "2022-08-17",                                                                                                                                                                              
  "repo": {                                                                                                                                                                                          
    "name": "github.com/azeemshaikh38/empty-repo-test",                                                                                                                                              
    "commit": "no commits found"                                                                                                                                                                     
  },                                                                                                                                                                                                 
  "scorecard": {                                                                                                                                                                                     
    "version": "v4.5.0-25-g887facf",                                                                                                                                                                 
    "commit": "887facf3cad2bd401ff5f6a51725225f8ec28c17"                                                                                                                                             
  },                                                                                                                                                                                                 
  "score": 5,                                                                                                                                                                                        
  "checks": [                                                                                                                                                                                        
     ...
   ]
}

The aggregate score of 5 for an empty repository does not make any sense and needs to be fixed.

@ossf/scorecard-maintainers @spencerschrock @raghavkaul fyi.

[azeemshaikh38]

One idea might be to have the Score field to be a pointer. That way a nil value can indicate NA instead of -1 which we today use to indicate inconclusive.

[leec94]

Hi, in testing #2158 I was testing scorecard on an empty repository, but got an empty result. It surprised me that even the repo name wasn't present in the results, so I wasn't sure if my github auth token was expired or wrong, so it led me astray refreshing my token before trying to run scorecard on another repo.

Suggestion, can the results still contain a valid date and repo fields in the results of an empty repo scan? -1 as the score is fine too. Fine with opening this up again or creating a new issue.

carolinelee@Carolines-MBP scorecard % go run main.go --repo=github.com/azeemshaikh38/empty-repo-test --format=json | jq .                
{
  "date": "0001-01-01T00:00:00Z",
  "repo": {
    "name": "",
    "commit": ""
  },
  "scorecard": {
    "version": "",
    "commit": ""
  },
  "score": -1,
  "checks": null,
  "metadata": null
}

[spencerschrock]

@leec94 would you like to send a PR, perhaps by implementing a new sentinel error instead of the empty commit semantics? (edit: although this may complicate things, one thing that I'd keep an eye out for is running scorecard on a local repo, and what it might look like there)

[leec94]

yes i'd like to take this on, feel free to assign me. thanks! do you have more details on the local repo part? so scanning scorecard is possible on local repos, is there more info on how and where that's done?

[spencerschrock]

yes i'd like to take this on, feel free to assign me. thanks! do you have more details on the local repo part? so scanning scorecard is possible on local repos, is there more info on how and where that's done?

It's a CLI flag that points to a directory, so something like:

$ go run main.go --local=. --format json | jq
{
  "date": "2023-08-23T15:44:20-07:00",
  "repo": {
    "name": "file://.",
    "commit": "unknown"
  },
  "scorecard": {
    "version": "",
    "commit": "unknown"
  },
  "score": 7.8,
  "checks": [
    {
      "details": null,
      "score": 0,
      "reason": "binaries present in source code",
      "name": "Binary-Artifacts",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#binary-artifacts",
        "short": "Determines if the project has generated executable (binary) artifacts in the source repository."
      }
    },

If you need to modify it during the PR, that code would live in https://github.com/ossf/scorecard/tree/d6ed8105df2354393f1678075a2848eac5cb71f4/clients/localdir