google
Full experiment with the new C-specific prompt
@DavidKorczynski: Are there any other changes required to use the C-specific prompt? Would you recommend updating the benchmarks before the exp?
@DavidKorczynski: Are there any other changes required to use the C-specific prompt?
No
Would you recommend updating the benchmarks before the exp?
Yeah, I don't think it makes too much sense to try on C++ unless we're testing that it doesn't break the workflow or so -- I wouldn't expect the generated prompts to generate good C++ harnesses. We can use these benchmarks for pure C projects only: https://github.com/google/oss-fuzz-gen/pull/371
The experiment report with c-benchmarks only: https://llm-exp.oss-fuzz.com/Result-reports/scheduled/2024-06-23-weekly-all/
~~## The incorrect header error persists.~~
~~Take bind9 as an example:
The prompt provided the header full path (/src/bind9/lib/dns/include/dns/view.h), but LLM still used the incorrect path (/src/bind9/lib/isc/include/dns/view.h), which highlights the need to fix this programmatically.
BTW, LLM generated the correct header with the default prompt.~~
~~I reckon to avoid regression, we can:~~
- ~~Only fix the header when compilation fails due to this.~~
- ~~Fix header programmatically: 1) Parse the wrong header from the error message, 2) find the correct file path, 3) replace the wrong line with the correct path, 4) compile the fuzz target again.~~
We are trying a different solution to this.
Missing function implementation
Here is an example.
@DavidKorczynski will you address these?
LLM code-bison-32k failed to generate results of some benchmarks because it is not available in certain regions.
Re-running under the same config as 2024-06-23-weekly-all/ with C benchmarks only using Gemini-1.5:
https://llm-exp.oss-fuzz.com/Result-reports/scheduled/2024-06-24-weekly-all/