Investigate Csmith for fuzzing xlscc

[meheff]

I wonder if Csmith would be useful for generating random input programs for fuzzing xlscc. It might need to be augmented with a wrapper which does I/O and feeds Csmith-generated functions. Not sure if Csmith would be able to generate ac_int types, however it probably would be good at stress testing other parts of xlscc such as control flow handling.

[ted-xie]

I very briefly took a stab at this and found a couple obstacles:

• csmith generates a lot of different types of code that I suspect are not supported by XLScc (or other commercial HLS C++ compilers, for that matters). Here are the flags I added to csmith to generate some sensical inputs for an HLS compiler: --no-argc --no-bitfields --no-volatiles --no-volatile-pointers --no-global-variables --nomain. I suspect more tweaking shall be necessary, e.g. to disallow overly liberal use of pointers.
• xlscc chokes when it encounters system header files such as #include <string.h> or #include <stdint.h>. I'm probably missing some xlscc flag but I couldn't identify which one(s) would resolve this issue. My xlscc command: xlscc kernel.c --top func_1 --include_dirs $HOME/sw/csmith/runtime --defines __SYNTHESIS__ .

[cdleary]

@ted-xie Did it feel like the obstacles could be handled with a rewrite / "early rejection script" pre-pass kind of thing?

(I imagine one of the challenges is that most of the interesting things are to do with ac_int types, as @meheff mentioned, and we won't be generating those.)

[ted-xie]

@cdleary XLScc was failing fast enough - iteration time was not the issue, but rather just getting the frontend to pass. I hadn't even gotten to the ac_int part yet.

My memory is already a little foggy on this since I looked at it 3 weeks ago, but IIRC the main issue was that the cmsith headers themselves would pull in system includes such as string.h. I think we might need some kind of basic postprocessor that dumps out just the kernel generated by CSmith that has as few dependencies as possible. Ideally we'd be able to generate the testbench (i.e. main()) and the kernel compilation unit separately. What do you think?

[ted-xie]

Alternatively, if there's an XLScc flag that tells the frontend to not choke on string.h, stdint.h etc that would be great too!

[ted-xie]

@JoshVarga @spurserh

Poked around with this a little more, couple updates:

• Recent updates to xlscc have added some basic system library support. There's basically enough to get a csmith-generated file to at least pass through the frontend, yay! I had to add a few extra goodies (symlink for string.h, stub files for stdio.h and limits.h) but things are looking good.
• Ended up adding a couple more args to csmith: --no-float --no-pointers . I think 1-level deep pointers are indeed valid C++ HLS input code for other tools, so I may eventually tweak --no-pointers to --max-pointer-depth=1.
• Had to include two extra command line options to xlscc to stub out incompatible coding styles in some of the csmith helper functions: -DUNSAFE -DUNSAFE_FLOAT (basically disables range checking for arithmetic ops, which should absolutely be enabled at least for integral types). These were disabled since they referenced macros typically defined in <limits.h>.

After all of the various hacks, we can get a first-pass reading on the csmith-generated file:

412 warnings generated.
UNIMPLEMENTED: For loop missing #pragma
   --> ./foo.cc:111:5
    |
111 |     for (i = 0; i < 5; i++)
    |     ^

command: bazel-bin/xls/contrib/xlscc/xlscc foo.cc --clang_args_file clang.args --top func_1

I hacked up some sed magic to automatically insert pragmas in front of for loops and ran into other issues.