multiplier icon indicating copy to clipboard operation
multiplier copied to clipboard

Published 20 hours ago • verified publisher icon trailofbits

Add fork+exec support, update Indexing.md, and handle "-mrelocation-model pic" flag

Open bsdb0y opened this issue 7 months ago • 5 comments

Summary This PR adds support for the --fork_mode flag in mx-index, which provides a fallback mechanism when --reproc_mode fails, particularly useful in Docker environments. This change addresses and resolves issue #577.

Usage mx-index --target compile_commands.json

Must specify --reproc_mode or --fork_mode.

Indexing can now be run with either of the following modes:

  • Using reproc:
    • mx-index --target compile_commands.json --reproc_mode --db sample.db
  • Using classic fork+exec as a fallback:
    • mx-index --target compile_commands.json --fork_mode --db sample.db

Additional Fix While generating compile_commands.json with clang, the flag -mrelocation-model pic also included and cause error. This error occurs because the pic part is mistakenly treated as a filename, resulting in errors like: Unable to find file pic An internal workaround has been added to handle this, although using gcc to generate the file avoids this issue entirely.

bsdb0y avatar May 09 '25 20:05 bsdb0y

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar May 09 '25 20:05 CLAassistant

Test1:

//cat sample2.c
#include <stdio.h>

// 1. Function to add two numbers
int add(int a, int b) {
    return a + b;
}

// 2. Function to find the maximum of two numbers
int max(int a, int b) {
    return (a > b) ? a : b;
}

// 3. Function to check if a number is even
int is_even(int n) {
    return (n % 2 == 0);
}

// 4. Function to print a message
void print_hello() {
    printf("Hello, World!\n");
}

// 5. Function to calculate factorial (non-recursive)
int factorial(int n) {
    int result = 1;
    for (int i = 2; i <= n; i++)
        result *= i;
    return result;
}

// Example usage
int main() {
    printf("Add: %d\n", add(5, 3));
    printf("Max: %d\n", max(10, 20));
    printf("Is 4 even? %d\n", is_even(4));
    print_hello();
    printf("Factorial of 5: %d\n", factorial(5));
    return 0;
}

reproc fails inside docker:

(install) root@b5158c58ce21:~/install# mx-index --target compile_commands.json --db sample2.db --reproc_mode
E20250510 02:38:48.142114  6108 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/bin/clang -c -g -O0 -o sample2 sample2.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:38:48.148221  6104 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/lib/llvm-20/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -dumpdir sample2- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -fdebug-compilation-dir=/root/install -fcoverage-compilation-dir=/root/install -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/15/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -O0 -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -x c -o /tmp/sample2-0daee7.o sample2.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable

(install) root@b5158c58ce21:~/install# ls -alSh sample2.db
-rw-r--r--. 1 root root 3.3M May 10 02:38 sample2.db

fork+exec works

(install) root@b5158c58ce21:~/install# mx-index --target compile_commands.json --db sample2.db --fork_mode

(install) root@b5158c58ce21:~/install# ls -alSh sample2.db
-rw-r--r--. 1 root root 3.4M May 10 02:38 sample2.db

(install) root@b5158c58ce21:~/install# mx-list-functions --db sample2.db
1152921504606847044     2305843009214745037     9281918834641469440     add     def
1152921504606847044     2305843009214745038     9281918834642518016     max     def
1152921504606847044     2305843009214745039     9281918834643566592     is_even def
1152921504606847044     2305843009214745040     9281918834644615168     print_hello     def
1152921504606847044     2305843009214745041     9281918834645663744     factorial       def
1152921504606847044     2305843009214745042     9281918834646712320     main    def

Test2: Real world application

  • command: --reproc_mode:
    • mx-index --db /root/OGHarn/demos/zlib/lib.db --target /root/OGHarn/demos/zlib/lib/compile_commands.json --workspace /root/OGHarn/demos/zlib/mx --reproc_mode

Failure logs inside docker:

E20250510 02:30:01.870612  5690 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -D_FILE_OFFSET_BITS=64 -o CMakeFiles/example64.dir/test/example.c.o -c /root/OGHarn/demos/zlib/lib/test/example.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.875167  5691 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -D_FILE_OFFSET_BITS=64 -o CMakeFiles/minigzip64.dir/test/minigzip.c.o -c /root/OGHarn/demos/zlib/lib/test/minigzip.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.883198  5709 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/zutil.c.o -c /root/OGHarn/demos/zlib/lib/zutil.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.891896  5704 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/minigzip.dir/test/minigzip.c.o -c /root/OGHarn/demos/zlib/lib/test/minigzip.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.899298  5706 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/example.dir/test/example.c.o -c /root/OGHarn/demos/zlib/lib/test/example.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.907377  5708 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/zutil.c.o -c /root/OGHarn/demos/zlib/lib/zutil.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.915544  5692 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/uncompr.c.o -c /root/OGHarn/demos/zlib/lib/uncompr.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.924079  5693 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/inflate.c.o -c /root/OGHarn/demos/zlib/lib/inflate.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.931890  5703 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/crc32.c.o -c /root/OGHarn/demos/zlib/lib/crc32.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.939175  5697 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/deflate.c.o -c /root/OGHarn/demos/zlib/lib/deflate.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.953272  5700 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/inftrees.c.o -c /root/OGHarn/demos/zlib/lib/inftrees.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.953497  5705 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/compress.c.o -c /root/OGHarn/demos/zlib/lib/compress.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.969448  5695 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/gzread.c.o -c /root/OGHarn/demos/zlib/lib/gzread.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.969564  5696 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/gzlib.c.o -c /root/OGHarn/demos/zlib/lib/gzlib.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.977866  5702 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/infback.c.o -c /root/OGHarn/demos/zlib/lib/infback.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.987006  5698 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/gzwrite.c.o -c /root/OGHarn/demos/zlib/lib/gzwrite.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:01.996883  5694 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/trees.c.o -c /root/OGHarn/demos/zlib/lib/trees.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.005138  5701 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/gzclose.c.o -c /root/OGHarn/demos/zlib/lib/gzclose.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.013847  5690 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/uncompr.c.o -c /root/OGHarn/demos/zlib/lib/uncompr.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.022713  5699 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/inffast.c.o -c /root/OGHarn/demos/zlib/lib/inffast.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.030362  5697 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/gzclose.c.o -c /root/OGHarn/demos/zlib/lib/gzclose.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.039072  5705 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/crc32.c.o -c /root/OGHarn/demos/zlib/lib/crc32.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.046243  5691 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/trees.c.o -c /root/OGHarn/demos/zlib/lib/trees.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.054956  5709 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/inffast.c.o -c /root/OGHarn/demos/zlib/lib/inffast.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.063756  5692 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/gzwrite.c.o -c /root/OGHarn/demos/zlib/lib/gzwrite.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.071182  5703 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/gzlib.c.o -c /root/OGHarn/demos/zlib/lib/gzlib.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.082054  5707 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -o CMakeFiles/zlibstatic.dir/adler32.c.o -c /root/OGHarn/demos/zlib/lib/adler32.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.090376  5708 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/inflate.c.o -c /root/OGHarn/demos/zlib/lib/inflate.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.100734  5700 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/deflate.c.o -c /root/OGHarn/demos/zlib/lib/deflate.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.109674  5693 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/gzread.c.o -c /root/OGHarn/demos/zlib/lib/gzread.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.118412  5704 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/inftrees.c.o -c /root/OGHarn/demos/zlib/lib/inftrees.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.127548  5695 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/compress.c.o -c /root/OGHarn/demos/zlib/lib/compress.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.138734  5706 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/infback.c.o -c /root/OGHarn/demos/zlib/lib/infback.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250510 02:30:02.148285  5696 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /root/AFLplusplus/afl-clang-fast -DZLIB_DLL -D_LARGEFILE64_SOURCE=1 -I/root/OGHarn/demos/zlib/lib -fPIC -o CMakeFiles/zlib.dir/adler32.c.o -c /root/OGHarn/demos/zlib/lib/adler32.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
  • command: --fork_mode:
    • mx-index --db /root/OGHarn/demos/zlib/lib.db --target /root/OGHarn/demos/zlib/lib/compile_commands.json --workspace /root/OGHarn/demos/zlib/mx --fork_mode which is success.

bsdb0y avatar May 09 '25 21:05 bsdb0y

LGTM!

pgoodman avatar May 12 '25 18:05 pgoodman

Ping @kumarak. I no longer have reviewer privileges here 😭.

pgoodman avatar May 12 '25 18:05 pgoodman

Ping @kumarak. I no longer have reviewer privileges here 😭.

Hi @kumarak, can you please take a look and approve the changes? Let me know if you need anything from me.

bsdb0y avatar May 18 '25 00:05 bsdb0y