rellic icon indicating copy to clipboard operation
rellic copied to clipboard

Add scripts for building/testing gcc torture tests

Open Toizi opened this issue 1 year ago • 4 comments

See #80

  • generate_compile_commands.py and compress_bitcode.sh have been taken from the AnghaBench repository and modified slightly for the gcc torture tests.
  • build_and_compress.sh will download the test suite using LLVM's release tags (e.g. "release/15.x") and call the other two scripts to build and compress the binaries/bitcode.
  • roundtrip.py has been modified to support recursing into subdirectories to support using it on the output generated by the lifting-tools-ci/tool_run_scripts/rellic.py script.
  • test-gcc-torture.sh is a modified version of test-angha-1k.sh with the option to run roundtrip.py on the output for the host architecture.
  • For the tests to work, the script lifting-tools-ci/datasets/fetch_gcc_torture.sh will need to be added analogous to the fetch_angha.sh one.

Only tested with LLVM 15 so far. Test output (slightly trimmed) using dummy fetch_gcc_torture.sh for just amd64

~/dev/rellic/rellic-gcc-torture ~/dev/rellic
bitcode.clang-15.amd64.tar.xz                                                                                                                                                                         100% 1410KB 180.4MB/s   00:00
[+] Testing architecture amd64
Running against Rellic:
rellic-decomp version unknown
Commit Hash: 87c50e8c89b5af9d00bc129d6c032849696422d5
Commit Date: 2022-12-22 13:37:23 +0100
Last commit by: Francesco Bertolaccini [[email protected]]
Commit Subject: [Migrate to LLVM15 (#308)]

Uncommitted changes were present during build.
Using LLVM 15.0.6


Listing files in /home/mg/dev/rellic/rellic-gcc-torture/bitcode/amd64
Found 1461 bitcode files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1461/1461 [08:50<00:00,  2.75it/s]
Run took 0:08:50.966609
Speed of 2.75 runs/sec
Success Metrics: [1377/1461]
Success Percentage: [94.25%]
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:380`: `4` failures
`ASTBuilder.cpp:158`: `4` failures
`timeout`: `4` failures
`ASTBuilder.cpp:413`: `1` failures
[!] There were [84] failures on amd64:
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:158`: `4` failures
`ASTBuilder.cpp:380`: `4` failures
[!] Failed decompilation for amd64
Running against Clang:
clang version 15.0.6
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/mg/dev/cxx-common/vcpkg/installed/x64-linux-rel/tools/llvm

Listing files in /home/mg/dev/rellic/rellic-gcc-torture/decompile/amd64
Found 1379 C files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1379/1379 [00:14<00:00, 95.92it/s]
Run took 0:00:14.381400
Speed of 95.89 runs/sec
Success Metrics: [1355/1379]
Success Percentage: [98.26%]
`use_of_undeclared_identifier`: `13` failures
`incomplete_definition_of_type`: `4` failures
`too_many_errors_emitted`: `3` failures
`expected_member_name_or`: `1` failures
`incompatible_pointer_to_integer`: `1` failures
`ISO_C_requires_a`: `1` failures
`passing_double`: `1` failures
[!] One or more failures encountered during test

Toizi avatar Jan 09 '23 20:01 Toizi

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Jan 09 '23 20:01 CLAassistant

@Toizi This is excellent!

pgoodman avatar Jan 09 '23 20:01 pgoodman

Hi @Toizi! This is really great. Can you share what your dummy fetch_gcc_torture.sh looks like?

frabert avatar Jan 18 '23 10:01 frabert

Hey @frabert, the dummy script will simply need to copy the compressed bitcode to the current directory. Since I've compiled it on a VM, my fetch_gcc_torture.sh was a simple one-liner: scp vmUbuntu22:dev/rellic/compiled/bitcode.clang-15.amd64.tar.xz .

Toizi avatar Jan 18 '23 11:01 Toizi