askama
askama copied to clipboard
djc
parser: add benchmark
Fuzzer:
cargo run --bin fuzz -r -- --progress
Benchmark:
cargo bench --bench min-size
Standalone parser:
cargo run --bin parse -- --input /some/file.jinja2
~~WIP:~~
- ~~not every node is implemented~~
- ~~there are some errors, either in the parser or the random source generator~~
I could only get honggfuzz to run properly. If I undid 5439c0b, it found the error quite quickly. It then dumps the input to arbitrary which can be translated to template source code with random_code/src/bin/translate-dump.rs.
If I activate instrumentation, libfuzzer always tells me
ERROR: The size of coverage PC tables does not match the
number of instrumented PCs. This might be a compiler bug,
please contact the libFuzzer developers.
Also check https://bugs.llvm.org/show_bug.cgi?id=34636
for possible workarounds (tl;dr: don't use the old GNU ld)
and I'm out of ideas. My normal link rustflags are -Clinker=clang++-18 -Clink-arg=-fuse-ld=lld-18, which might be too new. But downgrading to -17 or using gold did not help either.
I liked my homespun fuzzer better. :)
I think it's great to have a fuzzer but this is a lot of code to maintain. IMO we should have (a) a coverage-guided fuzzer and (b) it should work by taking random strings and trying to parse them, rather than painstakingly generating a whole bunch of combinations.
The code is mostly meant as a benchmarking tool. I found a few errors in the parser when I ran the benchmark, so I thought it might be useful for fuzzing, too. Sorry, I know too little about the topic and I have too much on my plate already to be able to read deeper into it. IMO the fuzz folder could be rm'd, and I'd scale the PR down to only benchmarking.
Sorry, I guess I misunderstood how this PR was put together. The current state of the PR still retains all the code to randomly generate templates, which is what I was objecting to in a previous comment, so I don't want to merge this as-is.
(Sorry to be dismissive of all the work you've done on this PR.)