lots of zombie processes?

[wideglide]

I think I'm running into issues where Angora might be failing because it is not reaping zombie child processes, filling up the process table, then unable to launch new processes. It appears that the fork server does read the status of the child processes, so there must be another invocation that doesn't check the exit codes?

Do you know where this might be originating from?

Here's an example with base64 from LAVA-M, the number of defunct processes just keeps growing over time.

<snip>
81689 ?        Zs     0:00 [base64] <defunct>
81708 ?        Zs     0:00 [base64] <defunct>
81738 ?        Zs     0:00 [base64] <defunct>
81757 ?        Zs     0:00 [base64] <defunct>
81778 ?        Zs     0:00 [base64] <defunct>
81794 ?        Zs     0:00 [base64] <defunct>
81808 ?        Zs     0:00 [base64] <defunct>
81842 ?        Zs     0:00 [base64] <defunct>
81898 ?        Zs     0:00 [base64] <defunct>
81900 ?        Zs     0:00 [base64] <defunct>
user@d-9-9-2:/dev/shm/fuzz/angora/7051337.7/who$ ps ax | grep '\[base64\] <defunct>'  | wc -l
1304

And here's the error log from another instance on the same host:

 WARN  angora::executor::forksrv  > Fail to read child_id -- Interrupted system call (os error 4)
 WARN  angora::executor::forksrv  > Fail to read child_id -- Interrupted system call (os error 4)
 WARN  angora::executor::forksrv  > Fail to read child_id -- Interrupted system call (os error 4)
 WARN  angora::executor::forksrv  > Fail to read child_id -- Interrupted system call (os error 4)
 ERROR angora::executor::forksrv  > FATAL: Failed to spawn child. Reason: Resource temporarily unavailable (os error 11)
thread '<unnamed>' panicked at 'explicit panic', fuzzer/src/executor/forksrv.rs:66:17
stack backtrace:
 WARN  angora::executor::forksrv  > Unable to request new process from frok server! -1
 ERROR angora::executor::forksrv  > FATAL: Failed to spawn child. Reason: Resource temporarily unavailable (os error 11)
thread '<unnamed>' panicked at 'explicit panic', fuzzer/src/executor/forksrv.rs:66:17
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:486
  11: std::panicking::begin_panic
  12: angora::executor::forksrv::Forksrv::new
  13: angora::executor::executor::Executor::rebind_forksrv
  14: angora::executor::executor::Executor::run
  15: angora::search::afl::AFLFuzz::run
  16: angora::fuzz_loop::fuzz_loop
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:486
  11: std::panicking::begin_panic
  12: angora::executor::forksrv::Forksrv::new
  13: angora::executor::executor::Executor::rebind_forksrv
  14: angora::executor::executor::Executor::run_with_cond
  15: angora::search::gd::GdSearch::cal_gradient
  16: angora::search::gd::GdSearch::run
  17: angora::fuzz_loop::fuzz_loop
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[wideglide]

Another instance of about 6K processes per fuzzing process

localuser@bot12b:~/archive/logs$ ps ax | wc -l
18278
localuser@bot12b:~/archive/logs$ ps ax | grep '\[duk\] <defunct>' | wc -l
5909
localuser@bot12b:~/archive/logs$ ps ax | grep '\[jq\] <defunct>' | wc -l
11817
localuser@bot12b:~/archive/logs$ ps ax | grep ' <defunct>' | wc -l
17727
localuser@bot12b:~/archive/logs$ pgrep -af angora/bin/fuzzer
4940 /angora/bin/fuzzer --sync_afl -i inputs -o outputs -t ./lava-ang/bin/jq.tt -j 2 --time_limit 9.0 -- ./lava-ang/bin/jq @@
30609 /angora/bin/fuzzer --sync_afl -i inputs -o outputs -t ./lava-ang/bin/jq.tt -j 2 --time_limit 9.0 -- ./lava-ang/bin/jq @@
31005 /angora/bin/fuzzer --sync_afl -i inputs -o outputs -t ./lava-ang/bin/duk.tt -j 2 --time_limit 2.0 -- ./lava-ang/bin/duk @@