Scorecard should provide details on fuzzing coverage in fuzzing check

[inferno-chromium]

@htuch's suggestion.

[htuch]

I think it would be good to understand fuzzer health overall. So, coverage is a big part, but so is performance, whether issues get fixed, other efficacy metrics that OSS-fuzz emits. If there was a way to capture this in simple A/B/C grades or the like it would be very useful. CC @asraa

[inferno-chromium]

@laurentsimon also mentioned this idea, adding him as fyi

[naveensrinivasan]

@inferno-chromium does oss-fuzz provide those details?

[asraa]

They're up on GCS for each project's latest coverage report e.g. https://oss-fuzz.com/coverage-report/job/libfuzzer_asan_abseil-cpp/latest

I'm not sure that's the easiest way to get them

[naveensrinivasan]

They're up on GCS for each project's latest coverage report e.g. https://oss-fuzz.com/coverage-report/job/libfuzzer_asan_abseil-cpp/latest

I'm not sure that's the easiest way to get them

This a good start. Thanks. I will work on updating the fuzzing with that information.

[naveensrinivasan]

They're up on GCS for each project's latest coverage report e.g. https://oss-fuzz.com/coverage-report/job/libfuzzer_asan_abseil-cpp/latest

I'm not sure that's the easiest way to get them

Is there a json result? I don't have permission to access the dir. We can use the HTTP endpoint to query the results and update the fuzzing results.

[naveensrinivasan]

@inferno-chromium / @asraa Ping.

[inferno-chromium]

@oliverchang can help on providing the json endpoint/gcs info here.

[oliverchang]

Sure!

So to get the coverage for a OSS-Fuzz project, you need to read 2 JSON files.

First from e.g. https://oss-fuzz-coverage.storage.googleapis.com/latest_report_info/woff2.json

Then get the "report_summary_path" from that. At time of writing this is "gs://oss-fuzz-coverage/woff2/reports/20210428/linux/summary.json" which can be converted to a HTTP link: https://oss-fuzz-coverage.storage.googleapis.com/woff2/reports/20210428/linux/summary.json

That JSON will have a "totals" field which gives coverage summaries. The best one to use will probably be the most fine grained one ("regions").

[naveensrinivasan]

The fuzzing results provide all these details.

type ossfuzz struct {
	Data []struct {
		Files []struct {
			Filename string `json:"filename"`
			Summary  struct {
				Branches struct {
					Count      int `json:"count"`
					Covered    int `json:"covered"`
					Notcovered int `json:"notcovered"`
					Percent    int `json:"percent"`
				} `json:"branches"`
				Functions struct {
					Count   int `json:"count"`
					Covered int `json:"covered"`
					Percent int `json:"percent"`
				} `json:"functions"`
				Instantiations struct {
					Count   int `json:"count"`
					Covered int `json:"covered"`
					Percent int `json:"percent"`
				} `json:"instantiations"`
				Lines struct {
					Count   int `json:"count"`
					Covered int `json:"covered"`
					Percent int `json:"percent"`
				} `json:"lines"`
				Regions struct {
					Count      int `json:"count"`
					Covered    int `json:"covered"`
					Notcovered int `json:"notcovered"`
					Percent    int `json:"percent"`
				} `json:"regions"`
			} `json:"summary"`
		} `json:"files"`
		Totals struct {
			Branches struct {
				Count      int     `json:"count"`
				Covered    int     `json:"covered"`
				Notcovered int     `json:"notcovered"`
				Percent    float64 `json:"percent"`
			} `json:"branches"`
			Functions struct {
				Count   int `json:"count"`
				Covered int `json:"covered"`
				Percent int `json:"percent"`
			} `json:"functions"`
			Instantiations struct {
				Count   int     `json:"count"`
				Covered int     `json:"covered"`
				Percent float64 `json:"percent"`
			} `json:"instantiations"`
			Lines struct {
				Count   int     `json:"count"`
				Covered int     `json:"covered"`
				Percent float64 `json:"percent"`
			} `json:"lines"`
			Regions struct {
				Count      int     `json:"count"`
				Covered    int     `json:"covered"`
				Notcovered int     `json:"notcovered"`
				Percent    float64 `json:"percent"`
			} `json:"regions"`
		} `json:"totals"`
	} `json:"data"`
	Type    string `json:"type"`
	Version string `json:"version"`
}

Questions

1. Should scorecard include all these details on the results? or will some section alone suffice?
2. The scorecard result does not have an option to include additional details like this by design.

type CheckResult struct {
	Error       error `json:"-"`
	Name        string
	Details     []string
	Confidence  int
	Pass        bool
	ShouldRetry bool `json:"-"`
}

We could extend the results by changing Details []string to Details interface{}. This will maintain backward compatibility on the json but I think it will break the BigQuery. This approach can also be used for the other checks if scorecard would like to extend the additional details.

[naveensrinivasan]

cc @azeemshaikh38

[inferno-chromium]

Totals -> Regions -> Percent should be good enough for now (high-level % of code covered). Additional details we can later add once we have some design on the Details string as you said. Right now, too many other things to worry about, but can come back in a month to create maybe some structure on Details field ? @azeemshaikh38 thoughts?

[azeemshaikh38]

+1 to adding this to Details []string for now. I think this is another of those things which will benefit from implementing https://github.com/ossf/scorecard/pull/347#discussion_r616364482.

We should probably consider updating how Details []string looks like after few weeks. But for now, there are too many moving parts and I don't want to add one more.

[naveensrinivasan]

We could wait for the Details results to be finalized. So that when we implement this without []string results.

If we implement with []string results and then move it when we finalize the design for results It will break the functionality.

@inferno-chromium Is this an issue that can wait ? are there teams waiting for this to be addressed?

[azeemshaikh38]

+1 to waiting if this is not a blocker.

[naveensrinivasan]

@azeemshaikh38 this would be a good feature to provide more details for package consumers.

Can the result interface accommodate some of these changes?

[laurentsimon]

Possible ideas:

1. Provide a percentage of code covered and provide a link to a detailed report (similar to Vulnerabilities check w/ OSV)? In this case no need to support the structure above
2. serialize the structure into a string, and set if to the details's Text or a new field. We can add a different Type check_result.go#L49
3. add an interface to LogMessage structure (with a new type) to support the structure.

(1) is the simplest to start with, and can be improved later to support a better solution.

[justaugustus]

Discussed in 5/16 meeting: nice to have, but unsure if this functionality could be extended to non-ossfuzz fuzzers