crash with subprocess not in container

[TomGoBravo]

I have a python job that uses starts ~8 sub processes. Sometimes py-spy exits with an error. The command line I run is: RUST_BACKTRACE=full py-spy record --output perf/master-a6df25b-a0ae67f.speedscope.json --format speedscope --rate 10 --subprocesses --native -- pyseir build-all -s IL --states-only --skip-download --output-dir output-root/master-a6df25b-a0ae67f

Here's the end of the output:

py-spy> Stopped sampling because process exitted
py-spy> Wrote speedscope file to 'perf/master-a6df25b-21b013d.speedscope.json'. Samples: 1548 Errors: 0
py-spy> Visit https://www.speedscope.app/ to view
Error: No child process (os error 10)
   0: failure::backtrace::internal::InternalBacktrace::new
   1: <failure::backtrace::Backtrace as core::default::Default>::default
   2: py_spy::main
   3: std::rt::lang_start::{{closure}}
   4: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
      std::panicking::try::do_call
             at src/libstd/panicking.rs:292
   5: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
   6: std::panicking::try
             at src/libstd/panicking.rs:270
      std::panic::catch_unwind
             at src/libstd/panic.rs:394
      std::rt::lang_start_internal
             at src/libstd/rt.rs:51
   7: main

$ py-spy --version
py-spy 0.3.3

[benfred]

Did the output get written successfully when the profiled process finished? Like is the problem that when the process exited py-spy returned an error on exit rather than success - rather than py-spy throwing an error when the process was still running?

[TomGoBravo]

IIRC the process exited, py-spy wrote the expected output, then py-spy seemed to exit with an error. It wasn't the first time py-spy exited with an error in similar circumstances, which is why I had RUST_BACKTRACE=full on.

[hermancollin]

First, thank you for this wonderful tool!

I'll add this: I ran across the same problem using py-spy to profile a python toolbox. Sometimes, when running the following:

py-spy record -sF -o <output_filename> -- <command>

I also get: Error: No child process (os error 10) It doesn't always happen e.g. when running this command 10 times in a row, it will throw the error 4-5 times. At the beginning, I thought it was an issue with subprocess tracking, but the same behavior happens when removing the -s flag.

At this point, because the output is actually written and the profiled process seems to have exited without failure, this seems like a minor bug. When running py-spy record directly from the CLI, this error really doesn't affect the workflow. However, in my case, I am using the profiler multiple times inside a long shell script called multiple times, so whenever the error is thrown, my script stops its execution. There are workarounds, but this behavior is definitely weird.

[hermancollin]

After further exploration, I realized the --nonblocking option completely solves the Error: No child process (os error 10) issue, so it could be related to the interaction between py-spy and the profiled program.

[benfred]

After further exploration, I realized the --nonblocking option completely solves the Error: No child process (os error 10) issue, so it could be related to the interaction between py-spy and the profiled program.

Interesting - that would indicate its probably related to the process locking code. @hermancollin do you have an example reproducing this? I'd like py-spy to exit cleanly

[hermancollin]

@benfred I am using py-spy 0.3.7 (also tried 0.4.0.dev1 as you suggested in another thread but same behavior). I am using it to profile a bunch of scripts from Spinal Cord Toolbox (see this for installation).

The simplest way to reproduce is to try the following command:

py-spy record -o test.svg -- sct_check_dependencies -short

If you try this a bunch of times, you will see that, sometimes, the error pops up and sometimes it doesn't.

Also the output of the forementioned profiled script is sometimes affected (not sure if this is normal, see below).

[Break-Neck]

Can confirm this error on py-spy 0.3.11.