serve icon indicating copy to clipboard operation
serve copied to clipboard

Add support for gRPC max connection age configuration

Open namannandan opened this issue 9 months ago • 0 comments

Description

Add support for server side gRPC max connection age configuration

References:

  1. Netty gRPC maxConnectionAge
  2. Netty gRPC maxConnectionAgeGrace
  3. Default gRPC KeepAlive configuration

Fixes #2999

Type of change

  • [x] New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

  • [ ] CI
    • https://github.com/pytorch/serve/blob/master/test/pytest/test_gRPC_inference_api.py
    • https://github.com/pytorch/serve/blob/master/test/pytest/test_gRPC_management_apis.py
  • [ ] Manual test
# gRPC management connection timeout is set to 5 seconds
$ cat custom_config.properties 
grpc_management_max_connection_age_ms=5000
grpc_management_max_connection_age_grace_ms=0

$ torchserve --ncs --start --model-store ./model_store --ts-config ./custom_config.properties

# Note: Model takes longer than 5 seconds to load
$ time python ts_scripts/torchserve_grpc_client.py register densenet161
## Check densenet161.mar in mar_set : set()
## Register marfile: https://torchserve.s3.amazonaws.com/mar_files/densenet161.mar

Failed to register model densenet161.
Socket closed
python ts_scripts/torchserve_grpc_client.py register densenet161  0.13s user 0.06s system 3% cpu 5.629 total

As can be seen from the test above, the gRPC management connection to register model is closed after the configured 5 seconds (max connection age).

namannandan avatar Apr 29 '24 19:04 namannandan