Measure thoroughness of glsl-fuzz on open source Vulkan targets via code coverage

[afd]

Background

The thoroughness of the shader compiler testing performed by GraphicsFuzz relates to:

a. The coverage achieved by the reference shaders used as a basis for fuzzing b. The further coverage achieved by transforming these reference shaders

For a given open source driver or open source shader processing tool, we would like to be able to answer questions such as:

1. What statement coverage do we get from running a particular reference shader?
2. What statement coverage do we get from running the set of all available reference shaders?
3. What statement coverage do we get from running at set of N variant shaders obtained from a single reference shader? (I.e. from one shader family.)
4. What statement coverage do we get from running a set of N variants shaders for every available reference shader? (I.e. from a full fuzzing run.)
5. What statement coverage does running dEQP achieve that is not achieved by the above, and vice versa?

Points 1 and 2 relate to (a) above - they help us identify where more shaders need to be added to the pool of references.

Points 3 and 4 related to (b) above - they help us understand where we need to look at adding richer transformations.

Point 5 helps to understand where there are gaps in dEQP (when fuzzing covers things that dEQP does not), and where there are gaps in the fuzzer (where dEQP covers things that fuzzing does not).

Requirement

Infrastructure and scripts to support asking questions 1--5 above, for SPIR-V shader compilation, with respect to:

• SwiftShader
• Mesa Vulkan targets for Intel and AMD GPUs
• spirv-opt
• glslang

Details

The current set of reference shaders we care about for this work are in:

graphicsfuzz/shaders/src/main/glsl/samples/310es/

To run one of these shaders under Vulkan:

graphicsfuzz-tool com.graphicsfuzz.generator.tool.PrepareReference --generate-uniform-bindings --max-uniforms 10 reference_shader_name.json prepared.json
runspv host prepared.json outdir # add ‘--spirvopt=-O’ or ‘--spirvopt=-Os’ to enable spirv-opt optimizations

To generate a variant from a reference and run it:

graphicsfuzz-tool com.graphicsfuzz.generator.tool.Generate --generate-uniform-bindings --max-uniforms 10 reference_shader_name.json /path/to/graphicsfuzz/shaders/src/main/glsl/samples/310es variant.json
runspv host variant.json outdir # again, add ‘--spirvopt=-O’ or ‘--spirvopt=-Os’ to enable spirv-opt optimizations

[afd]

A few notes on this:

• Coverage information being gathered using gcov / lcov would make sense

• Needs to work under Linux (for Mesa drivers)

• Getting coverage of SwiftShader running on some subset of dEQP tests would be a good starting point

• We want to ask coverage difference questions, so having coverage of SwiftShader on two sets of tests, A and B, and seeing the difference between coverage(A) and coverage(B) would be a good next step

[paulthomson]

Jussi has made progress on this. He used the CMake CodeCoverage extension. Here are some notes for another way that I tried. I might update it.

git clone https://swiftshader.googlesource.com/SwiftShader

cd SwiftShader

mkdir -p out/build
cd out/build

BUILD_DIR=$(pwd)

export CFLAGS=--coverage
export CXXFLAGS=--coverage
export LDFLAGS=--coverage

cmake -G Ninja ../.. -DCMAKE_BUILD_TYPE=Debug
cmake --build . --config Debug

export VK_ICD_FILENAMES=$BUILD_DIR/Linux/vk_swiftshader_icd.json

cd ../..

mkdir -p out/cov
cd out/cov

# Without this, the coverage files are output in out/build, alongside the object files.
# This seems to work well, but here is way to avoid this.
# export GCOV_PREFIX=$(pwd)/cov

amber -d simple.shader_test

lcov --capture  --directory $BUILD_DIR --output-file coverage.info
genhtml coverage.info --output-directory out

Running amber again adds to the existing coverage data.

[asuonpaa]

To clear counters between test runs use lcov --zerocounters --directory $BUILD_DIR or use gcovr with -d option to clean up coverage data after report generation.

[paulthomson]

We have a coverage report to go through. Please see instructions here.