relay
relay copied to clipboard
Implement graceful shutdown
Follow-up to https://github.com/getsentry/team-ingest/issues/345: We want to give Relay plenty of time to burn down an envelope buffer backlog after it has received the SIGTERM shutdown signal, so we should set a long grace period.
Unfortunately Relay currently never shuts down before the end of the grace period, which prevents us from setting a longer grace period. To allow Relay to terminate when it is "done",
- Stop accepting envelopes after the shutdown signal has been received,
- ensure that all services subscribe to the shutdown signal,
- break any communication cycles between services. Cycles might be fine during normal operations but could extend shutdown indefinitely).
- Ensure that all services break out of their main loops once they do not receive any new data.
- In
relay_server::run, do not awaitController::shutdown_handle().finished(). Instead, await a join handle for each service.
Relates to https://github.com/getsentry/relay/issues/4037, https://github.com/getsentry/relay/pull/4026 (join handles).
See summary of discussion here.