centraldogma icon indicating copy to clipboard operation
centraldogma copied to clipboard

`CentralDogma` may incorrectly report status as healthy when shutting down

Open jrhee17 opened this issue 4 years ago • 0 comments

Currently, CentralDogma shuts down resources in the following order:

  1. sessionManager
  2. file channels
  3. executors
  4. the server

https://github.com/line/centraldogma/blob/e93bc5a04eb3d38a0f315d9c5fdb0db3b0af1f52/server/src/main/java/com/linecorp/centraldogma/server/CentralDogma.java#L828-L832

As more long running tasks/repositories are added to CentralDogma (either from the API or via plugins), it is more likely that shutting down will take more time at intermediate steps.

When CentralDogma starts shutting down the actual server, health check will be turned off and the server will stop receiving requests.

https://github.com/line/centraldogma/blob/e93bc5a04eb3d38a0f315d9c5fdb0db3b0af1f52/server/src/main/java/com/linecorp/centraldogma/server/CentralDogma.java#L904

We may do one of the following:

  1. try to shut down the server before cleaning up internal resources
  2. set the server status as unhealthy before starting to shut down resources

jrhee17 avatar Jan 06 '22 00:01 jrhee17