golem icon indicating copy to clipboard operation
golem copied to clipboard

Golem standalone server doesn't catch SIGINT

Open danieletorelli opened this issue 7 months ago • 3 comments

A running instance of the Golem standalone server doesn't seem to catch SIGINT termination signal (e.g. pressing Ctrl+C in the running terminal window or using kill -INT <pid>).

danieletorelli avatar May 17 '25 21:05 danieletorelli

Interesting, it always works for me, if you mean golem server run

vigoo avatar May 17 '25 22:05 vigoo

Just running an idle server does seems to respond. i suspect there might be still some cases where we accidentally block, and maybe in those cases singal processing is simply starving.

If you run into this again please share the component / previous CLI calls and invokes.

noise64 avatar May 19 '25 17:05 noise64

The way I can always reproduce this is via a dockerized Golem server, so I suspect this is not working on Linux. Both Ctrl+C in terminal or stopping the container should send a SIGINT but without effect. This happens even without sending any golem command before.

Example:

~ % docker run -it --rm -p 9006:9006 -p 9881:9881 --name golem golem
Selected profile: local
  2025-05-20T04:22:44.345100Z  INFO opentelemetry_sdk: , name: "MeterProvider.Built"
    at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.28.0/src/metrics/meter_provider.rs:306

[... truncated output ...]

  2025-05-20T04:22:44.519046Z  INFO golem_common::retries: op success, duration_ms: 136, target_label: "worker_executor", op_label: "assign_shards", op_id: "a75812df5364:40679 (127.0.0.1)"
    at /cargo/git/checkouts/golem-f136eb08d75a2850/0a04f54/golem-common/src/retries.rs:159
    in golem_common::retries::retry with target: "worker_executor", op: "assign_shards", op_id: "a75812df5364:40679 (127.0.0.1)", attempt: 1

Then:

% docker exec -it golem ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.5  0.8 2403984 87296 pts/0   Ssl+ 04:22   0:00 golem server run -vv

And:

~ % docker exec -it golem kill -INT 1

Produces no effect on the running server:

~ % docker exec -it golem ps aux     
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.4  0.9 2403984 91776 pts/0   Ssl+ 04:22   0:00 golem server run -vv

danieletorelli avatar May 20 '25 04:05 danieletorelli