rellic
rellic copied to clipboard
Add scripts for building/testing gcc torture tests
See #80
-
generate_compile_commands.py
andcompress_bitcode.sh
have been taken from theAnghaBench
repository and modified slightly for the gcc torture tests. -
build_and_compress.sh
will download the test suite using LLVM's release tags (e.g. "release/15.x") and call the other two scripts to build and compress the binaries/bitcode. -
roundtrip.py
has been modified to support recursing into subdirectories to support using it on the output generated by thelifting-tools-ci/tool_run_scripts/rellic.py
script. -
test-gcc-torture.sh
is a modified version oftest-angha-1k.sh
with the option to runroundtrip.py
on the output for the host architecture. - For the tests to work, the script
lifting-tools-ci/datasets/fetch_gcc_torture.sh
will need to be added analogous to the fetch_angha.sh one.
Only tested with LLVM 15 so far.
Test output (slightly trimmed) using dummy fetch_gcc_torture.sh
for just amd64
~/dev/rellic/rellic-gcc-torture ~/dev/rellic
bitcode.clang-15.amd64.tar.xz 100% 1410KB 180.4MB/s 00:00
[+] Testing architecture amd64
Running against Rellic:
rellic-decomp version unknown
Commit Hash: 87c50e8c89b5af9d00bc129d6c032849696422d5
Commit Date: 2022-12-22 13:37:23 +0100
Last commit by: Francesco Bertolaccini [[email protected]]
Commit Subject: [Migrate to LLVM15 (#308)]
Uncommitted changes were present during build.
Using LLVM 15.0.6
Listing files in /home/mg/dev/rellic/rellic-gcc-torture/bitcode/amd64
Found 1461 bitcode files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1461/1461 [08:50<00:00, 2.75it/s]
Run took 0:08:50.966609
Speed of 2.75 runs/sec
Success Metrics: [1377/1461]
Success Percentage: [94.25%]
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:380`: `4` failures
`ASTBuilder.cpp:158`: `4` failures
`timeout`: `4` failures
`ASTBuilder.cpp:413`: `1` failures
[!] There were [84] failures on amd64:
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:158`: `4` failures
`ASTBuilder.cpp:380`: `4` failures
[!] Failed decompilation for amd64
Running against Clang:
clang version 15.0.6
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/mg/dev/cxx-common/vcpkg/installed/x64-linux-rel/tools/llvm
Listing files in /home/mg/dev/rellic/rellic-gcc-torture/decompile/amd64
Found 1379 C files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1379/1379 [00:14<00:00, 95.92it/s]
Run took 0:00:14.381400
Speed of 95.89 runs/sec
Success Metrics: [1355/1379]
Success Percentage: [98.26%]
`use_of_undeclared_identifier`: `13` failures
`incomplete_definition_of_type`: `4` failures
`too_many_errors_emitted`: `3` failures
`expected_member_name_or`: `1` failures
`incompatible_pointer_to_integer`: `1` failures
`ISO_C_requires_a`: `1` failures
`passing_double`: `1` failures
[!] One or more failures encountered during test
@Toizi This is excellent!
Hi @Toizi! This is really great. Can you share what your dummy fetch_gcc_torture.sh
looks like?
Hey @frabert, the dummy script will simply need to copy the compressed bitcode to the current directory.
Since I've compiled it on a VM, my fetch_gcc_torture.sh
was a simple one-liner:
scp vmUbuntu22:dev/rellic/compiled/bitcode.clang-15.amd64.tar.xz .