relay icon indicating copy to clipboard operation
relay copied to clipboard

Implement graceful shutdown

Open jjbayer opened this issue 1 year ago • 0 comments

Follow-up to https://github.com/getsentry/team-ingest/issues/345: We want to give Relay plenty of time to burn down an envelope buffer backlog after it has received the SIGTERM shutdown signal, so we should set a long grace period.

Unfortunately Relay currently never shuts down before the end of the grace period, which prevents us from setting a longer grace period. To allow Relay to terminate when it is "done",

  1. Stop accepting envelopes after the shutdown signal has been received,
  2. ensure that all services subscribe to the shutdown signal,
  3. break any communication cycles between services. Cycles might be fine during normal operations but could extend shutdown indefinitely).
  4. Ensure that all services break out of their main loops once they do not receive any new data.
  5. In relay_server::run, do not await Controller::shutdown_handle().finished(). Instead, await a join handle for each service.

Relates to https://github.com/getsentry/relay/issues/4037, https://github.com/getsentry/relay/pull/4026 (join handles).

See summary of discussion here.

jjbayer avatar Sep 20 '24 06:09 jjbayer