Umbrella issue: Projects failing

[DavidKorczynski]

Umbrella issue for projects failing.

• bitcoin-core issue
• nss.
  • ~~Compilation problem~~ problem identified and fix is coming up. The problem was resource exhaustion, we simply need to limit resources in the Docker container.

[DavidKorczynski]

@Navidem I just tried nss and it killed my machine - most likely due to some resource exhaustion but am yet to isolate whether it's due to LTO or fuzz-introspector, or something third

[Navidem]

@DavidKorczynski thanks for checking. I can say without LTO it finishes successfully within ~7 min on my side.

[DavidKorczynski]

@DavidKorczynski thanks for checking. I can say without LTO it finishes successfully within ~7 min on my side.

I too compile it fine with python3 infra/helper.py build_fuzzers nss

[DavidKorczynski]

I can now build NSS with fuzz-introspector:

The issue is that NSS uses a lot of processes during compilation and each of these also uses a lot of memory. This can be solved by changing this line https://hg.mozilla.org/projects/nss/file/tip/build.sh#l276 from run_scanbuild "$ninja" -C "$target_dir" "${ninja_params[@]}" to run_scanbuild "$ninja" -C "$target_dir" -j 2 "${ninja_params[@]}"

I will explore if we can use some constraints on the Docker environment, e.g. https://docs.docker.com/config/containers/resource_constraints/#cpu This should work in a general way.

[oliverchang]

@Navidem Let's create sub bugs and track them here for the ones you notice too!

[Navidem]

freeimage fails to compile. Apparently it is not compatible with -flto.

[Navidem]

Division by zero in post-processing/main.py These projects encounter div by zero once the profile is not produced or fail to merge:

• croaring [update: fixed]
• nettle [update: fixed]
• abseil-cpp [update: fixed]

Sample output for abseil-cpp:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating project profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating merged profile of 0 profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions reached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions unreached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating all_functions dictionary
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Gathering complexity and incoming references of each function
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Completed creationg of merged profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Refining profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating reachability overview table
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 44, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     fuzz_html.create_html_report(
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 526, in create_html_report
Step #6 - "compile-libfuzzer-introspector-x86_64":     html_string += create_top_summary_info(tables, project_profile)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 261, in create_top_summary_info
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_percentage = ((len(total_reached_functions)*1.0) / (len(total_reached_functions) + len(
Step #6 - "compile-libfuzzer-introspector-x86_64": ZeroDivisionError: float division by zero

Sample output for nettle:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-1.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-0.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 2 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-1.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating project profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating merged profile of 0 profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions reached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions unreached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating all_functions dictionary
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Gathering complexity and incoming references of each function
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Completed creationg of merged profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Refining profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating reachability overview table
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 44, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     fuzz_html.create_html_report(
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 526, in create_html_report
Step #6 - "compile-libfuzzer-introspector-x86_64":     html_string += create_top_summary_info(tables, project_profile)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 261, in create_top_summary_info
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_percentage = ((len(total_reached_functions)*1.0) / (len(total_reached_functions) + len(
Step #6 - "compile-libfuzzer-introspector-x86_64": ZeroDivisionError: float division by zero

[Navidem]

The following projects fail to parse/merge profile because of encoding issues:

• kamailio [update: fixed]

Sample output for kamailio:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-1.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-0.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 2 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-1.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzz_parse_msg.covreport -- matches regex: .*\.covreport$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzz_uri.covreport -- matches regex: .*\.covreport$
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 34, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     profile.accummulate_profile(target_folder)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_data_loader.py", line 196, in accummulate_profile
Step #6 - "compile-libfuzzer-introspector-x86_64":     self.load_coverage(target_folder)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_data_loader.py", line 125, in load_coverage
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_hit, coverage_map = fuzz_cov_load.llvm_cov_load(target_folder, self.get_target_fuzzer_filename())
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_cov_load.py", line 50, in llvm_cov_load
Step #6 - "compile-libfuzzer-introspector-x86_64":     for line in pf:
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/codecs.py", line 322, in decode
Step #6 - "compile-libfuzzer-introspector-x86_64":     (result, consumed) = self._buffer_decode(data, self.errors, final)
Step #6 - "compile-libfuzzer-introspector-x86_64": UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3854: invalid start byte

[DavidKorczynski]

Division by zero in post-processing/main.py These projects encounter div by zero once the profile is not produced or fail to merge:

~~Div by zero fixed in https://github.com/ossf/fuzz-introspector/commit/b03c4cfcfbec51779133adf24d14d78425e9121c~~ <-- This fixed another issue. The div-by-zero you highlight has not been fixed yet.

[DavidKorczynski]

https://github.com/ossf/fuzz-introspector/issues/30#issuecomment-1028514294

Kamailio should be fixed with https://github.com/ossf/fuzz-introspector/commit/554ae742c4d576da2b6334e132cda1ad6e016316

have run locally where I verified the issue and also verified post-fix that it works now

[DavidKorczynski]

Division by zero in post-processing/main.py These projects encounter div by zero once the profile is not produced or fail to merge:

• croaring
• nettle
• abseil-cpp

This issue seem to have happened because these lines in the OSS-Fuzz integration:

https://github.com/google/oss-fuzz/blob/cb45dab05cdd76d34e2f16cdd96a1c1c1d12d701/infra/base-images/base-builder/compile#L210-L211

The problem is those find commands can overwrite fuzzer profiles if the fuzzer profiles have the same name. Fuzzer profiles will often have the same filename but in different locations.

I fixed this by adding a UUID to the fuzzer log files now: https://github.com/ossf/fuzz-introspector/commit/4231c270ce6d8750b1b774f1f2ddca74f5c40d22 and I have verified that both croaring and nettle works now. As such, it shouldn't really happen now, but, the two lines I link to in the OSS-Fuzz integration should probably change considering fuzz introspector does not guarantee uniqueness (it almost does not) in output files. Fuzz introspector output files does guarantee {path, filename} to be unique though.

[Navidem]

The following projects were failing because of keyError in all_functions dict, fixed by 65beb16ee75c071c88a0602b27f9f09cfb023d4f and 664f3bf36007b3525150a59a9b806c605be38616

• orbit
• tinygltf
• espeak-ng
• snappy
• netcdf
• wuffs
• dropbear

[Navidem]

The following projects are failing with configure error Unknown float word ordering.

• mercurial
• python3-libraries
• cpython3

A quick search shows people recommending use of -ffat-lto-objects when compiling for lto. Have not yet tested.

[DavidKorczynski]

I added a small script to automate testing projects of oss-fuzz projects here https://github.com/ossf/fuzz-introspector/blob/main/oss_fuzz_integration/test_projects.sh

[Navidem]

I did an investigation and extracted projects that are failing to link/build fuzzers and excluded those that are also failing with coverage build. Furthermore, I filtered those that fail when we only enable lto. These were projects that fail once -flegacy-pass-manager -flto flag is passed (without introspector pass):

bignum-fuzzer
freeradius
gnutls
cairo
ffmpeg
gdal
cpp-httplib
curl
freeimage
ghostscript
grok
hiredis
hoextdown
icu
jansson
lame
libidn2
libphonenumber
libpsl
librdkafka
libteken
libvips
llhttp
mdbtools
mpg123
ndpi
num-bigint
opencv
openh264
openssh
osquery
poppler
qpdf
rustcrypto
samba
sqlite3
tesseract-ocr
unrar
wget
wget2
wxwidgets
xnu
xvid

These are the project that we may give lower priority to debug. On the other hand, the followings are the projects that introspector's pass is the main cause of build failure:

bloaty
dropbear
freetype2
gdk-pixbuf
glib
gpac
gstreamer
irssi
kcodecs
keystone
lcms
libprotobuf-mutator
mupdf
pidgin
quickjs
solidity
wuffs
xmlsec

[DavidKorczynski]

introspector's pass is the main cause of build failure:

Do you mean here it's specifically to compiler pass or do some of these projects fail in the post-processing?

[DavidKorczynski]

quickjs is working for me

[DavidKorczynski]

wuffs is working for me, although the reports have fairly limited calltrees, and I think this is because of use of indirect pointers, e.g.: https://github.com/google/wuffs/blob/ccbec1833094b1d2ee18b969aaae356a46bc5b30/internal/cgen/base/pixconv-submodule.c#L5270-L5281

Ref https://github.com/ossf/fuzz-introspector/issues/4

[DavidKorczynski]

Are we sure that all of these projects are related to fuzz-introspector? I am asking this because I debugged an issue over here https://github.com/ossf/fuzz-introspector/issues/48#issuecomment-1063163173 where it seems that the issue only happens when compiling with --sanitizer=introspector, however, the issue occurs even if FUZZ_INTROSPECTOR is unset, meaning that fuzz-introspector is essentially not run.

One of the potential issues is that --sanitizer=introspector also applies some other changes, such as specify

export LDFLAGS="-fuse-ld=gold"
export AR=llvm-ar
export RANLIB=llvm-ranlib

We should check if projects fail with these set, or, more specifically, if projects fail with --sanitizer=introspector but where FUZZ_INTROSPECTOR is unset. Because of they fail with that it's really not fuzz-introspector code as such.

I recon it would be nice to have some infrastructure to make it easy to isolate whether a given issue happens because of the fuzz-introspector llvm pass or e.g. the fact that we force use of -fuse-ld=gold. Also, it may seem that some issues only happens in the OSS-Fuzz environment, e.g. jsoncpp (see https://github.com/ossf/fuzz-introspector/issues/48#issuecomment-1063163173).

I am currently looking at curl where I am suspecting the issue doesn't happen because of fuzz-introspector code but rather switchig to a linker that is not supported by curl. This infrastructure might be included here https://github.com/ossf/fuzz-introspector/issues/41

[Navidem]

cleared v1 milestone, as the debugging remaining failures has to continue afterwards.