scorecard-action
scorecard-action copied to clipboard
ossf
CI Test report does not match the result from scorecard
The scorecard action is reporting that most of the PRs are not running tests even though running scorecard command reports that 30/30 tests ran tests correctly:
Image from security tab report as generated by scorecard action:
Data from running the check with the same PAT as the action:
docker run -e GITHUB_AUTH_TOKEN=<pat> gcr.io/openssf/scorecard:v4.1.0@sha256:a1e9bb4a0976e800e977c986522b0e1c4e0466601642a84470ec1458b9fa6006 --show-details --repo=https://github.com/flutter/flutter --verbosity=debug --checks=CI-TESTS
Starting [CI-Tests]
RESULTS
-------
Aggregate score: 10.0 / 10
Check scores:
Finished [CI-Tests]
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| SCORE | NAME | REASON | DETAILS | DOCUMENTATION/REMEDIATION |
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| 10 / 10 | CI-Tests | 30 out of 30 merged PRs | Debug: CI test found: pr: 101645, context: flutter-dashboard: | https://github.com/ossf/scorecard/blob/33f80c93dc79f860d874857c511c4d26d399609d/docs/checks.md#ci-tests |
| | | checked by a CI test -- score | https://api.github.com/repos/flutter/flutter/check-runs/5956689942 | |
| | | normalized to 10 | Debug: CI test found: pr: 101641, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5955662569 | |
| | | | Debug: CI test found: pr: 101638, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5955114082 | |
| | | | Debug: CI test found: pr: 101634, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5954663793 | |
| | | | Debug: CI test found: pr: 101625, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5953234632 | |
| | | | Debug: CI test found: pr: 101200, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/f5a3c19f2ea7b765e5892d5c56535fa94ef8b62e | |
| | | | Debug: CI test found: pr: 101619, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5952277466 | |
| | | | Debug: CI test found: pr: 101616, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5951947232 | |
| | | | Debug: CI test found: pr: 101613, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5951119409 | |
| | | | Debug: CI test found: pr: 101612, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5892676941 | |
| | | | Debug: CI test found: pr: 101549, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5881408311 | |
| | | | Debug: CI test found: pr: 100794, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/d8435ca1e908b937c45ed54ba08bf97dc3312a1d | |
| | | | Debug: CI test found: pr: 100787, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/cd4acce2deecb8c83a434d0742725d1d19900493 | |
| | | | Debug: CI test found: pr: 101607, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5892004554 | |
| | | | Debug: CI test found: pr: 101592, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/531fd1f2feefe3e6c62d1e60b7a501f56cde8daa | |
| | | | Debug: CI test found: pr: 98549, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/71de516c5766c7aa24280062b81cc6e6ac5cf3c2 | |
| | | | Debug: CI test found: pr: 100893, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5720651123 | |
| | | | Debug: CI test found: pr: 101600, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/742570db066efab2bb914cfb7ddd2022f9d9c50a | |
| | | | Debug: CI test found: pr: 101537, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5877395802 | |
| | | | Debug: CI test found: pr: 101567, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/f84cd8f66c6e90fc0bf6d62e1e7c7dda310cce8e | |
| | | | Debug: CI test found: pr: 101554, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5886492163 | |
| | | | Debug: CI test found: pr: 101583, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5887260706 | |
| | | | Debug: CI test found: pr: 101544, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5877504384 | |
| | | | Debug: CI test found: pr: 101553, context: Google testing: | |
| | | | https://api.github.com/repos/flutter/flutter/statuses/3d3a80ac3e530b93f6a1346d15f76d0cce1d0fb8 | |
| | | | Debug: CI test found: pr: 101564, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5881851413 | |
| | | | Debug: CI test found: pr: 101562, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5881102473 | |
| | | | Debug: CI test found: pr: 101559, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5880429584 | |
| | | | Debug: CI test found: pr: 101550, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5878652628 | |
| | | | Debug: CI test found: pr: 101548, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5878154047 | |
| | | | Debug: CI test found: pr: 101545, context: flutter-dashboard: | |
| | | | https://api.github.com/repos/flutter/flutter/check-runs/5877594658 | |
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
Interesting. Not sure what's going on here. I think we need a way to add debug input to the output. Created https://github.com/ossf/scorecard-action/issues/176 for tracking
The alert information is included in the SARIF file, is caching used somewhere in between calling scorecards and the generation of the SARIF file?
There's no caching. We always run scorecard and create the SARIF for each run.
It replicates with gcr.io/openssf/scorecard@sha256:8165ad910019422f40c51cbb97ff6e7db0e2e2e11faebf59e0b6f1a2eb66ebd7 but not with the latest images. Seems like it will also get fixed with the next update.