infra icon indicating copy to clipboard operation
infra copied to clipboard

Spawning sandboxes gets stuck on loading snapshot

Open ValentaTomas opened this issue 1 year ago • 1 comments

Sometimes when spawning a sandbox the process gets stuck on loading the snapshot and times out.

One of the possible causes of this is the network namespace handling in Go — it is possible that the goroutine namespace could be somehow switched because of the way the threads and namespaces are handled.

This bug would cause the request for sandbox to return 500 "Cannot create a environment instance right now" error.

ValentaTomas avatar Apr 02 '24 01:04 ValentaTomas

I think this is caused by the UFFD handler that sometimes panics with:

uffd_msg not ready
stack backtrace:
   0: rust_begin_unwind
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:595:5
   1: core::panicking::panic_fmt
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:67:14
   2: core::panicking::panic_display
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:150:5
   3: core::panicking::panic_str
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:134:5
   4: core::option::expect_failed
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/option.rs:1988:5
   5: uffd_valid_handler::uffd_utils::Runtime::run
   6: uffd_valid_handler::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at src/firecracker/examples/uffd/valid_handler.rs:32:14:

ValentaTomas avatar May 23 '24 10:05 ValentaTomas

This issue is no longer relevant as we moved the UFFD handler into the orchestrator.

ValentaTomas avatar Jan 07 '25 02:01 ValentaTomas