Caliptra Input fuzzing

[jhand2]

We should do some fuzzing of Caliptra's interface to the SoC (Mailbox commands, registers).

In particular we want to check for hangs/panics that can be caused externally.

[MDr164]

Quick question regarding the way this input should be generated. Is planned to use tools like cargo-fuzz for generating initial fuzzing input on the firmware side of things or is it planned to create fuzz input from the simulator to test the input directly via those registers aka validating both firmware and the transport mechanism used to supply external input?

[jhand2]

I think the higher fidelity option would be to try to fuzz the registers externally. This falls into 3 possible buckets:

1. Use the sw-simulator. This is better than nothing, but it would be better to use the real hw model. Benefit is this is fast. But we may also run into cases where the simulation isn't quite right and there are false positives. Maybe that's ok.
2. Use the verilator model. This is an actual model of the RTL, so fidelity is much higher. But it is very slow. Maybe it would make sense to do development of the fuzzer against the sw model, and then run in over days against verilator to try to catch issues.
3. Some folks are working on getting FPGA set-ups, but this is still in-progress.

cc @korran for any additional input

[korran]

Fuzzing is most useful when it can be profile-guided, which is easiest for unit tests that can be compiled for the host environment, or (more difficult) running in an emulator that can produce execution traces the fuzzer can use.

[korran]

If my goal was to fuzz-test the entire firmware, I would probably try to create a fake implementation of the Mmio trait that uses code from sw-emulator to implement the peripheral registers, then compile the whole thing (firmware + fake peripherals) for x86 with fuzzing instrumentation.