azure-functions-java-worker icon indicating copy to clipboard operation
azure-functions-java-worker copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Gracefully shutdown worker on WorkerTerminate request

Open shreyas-gopalakrishna opened this issue 2 years ago • 2 comments

Issue describing the changes in this PR

resolves #630 https://github.com/Azure/azure-functions-host/issues/2308 https://github.com/Azure/azure-functions-host/issues/3296

attached design doc: https://dev.azure.com/msazure/One/_git/AAPT-Antares-Docs?path=/TeamDocs/FunctionTeamDocs/Design/OOPWorkers/GracefulShutdownOfWorker.md&_a=preview&anchor=out-of-proc-support-for-cancellation-tokens

The flow would be the following:

  • If the worker does not support termination notification (based on capabilities RpcWorkerConstants.HandlesWorkerTerminateMessage), no change in the behavior.
  • Otherwise; the host will send a termination message and wait for the process to exit up to the timeout for process termination.
  • The worker's responsibility would be to handle that message and exit the process within the allotted time. In the case of .NET, application lifetime services will be notified and if customers are subscribing to shutdown events and tokens, they would also receive the notification. This would also terminate hosted services.
  • If the timeout is hit, the process would be killed.

Pull request checklist

  • [x] My changes do not require documentation changes
    • [ ] Otherwise: Documentation issue linked to PR
  • [ ] My changes should not be added to the release notes for the next release
    • [x] Otherwise: I've added my notes to release_notes.md
  • [ ] My changes do not need to be backported to a previous version
    • [ ] Otherwise: Backport tracked by issue/PR #issue_or_pr
  • [ ] I have added all required tests (Unit tests, E2E tests)

shreyas-gopalakrishna avatar Aug 18 '22 19:08 shreyas-gopalakrishna

Hi @liliankasem , just to confirm that host will restart after it send out WorkerTerminate request to language worker right. Do we long how many time we got between WorkerTerminate request send out and host restart? Thanks.

kaibocai avatar Aug 19 '22 14:08 kaibocai

Hi @liliankasem , just to confirm that host will restart after it send out WorkerTerminate request to language worker right. Do we long how many time we got between WorkerTerminate request send out and host restart? Thanks.

Grace period is set here, which I believe is currently 5 seconds @surgupta-msft can confirm

liliankasem avatar Aug 19 '22 16:08 liliankasem