serve
serve copied to clipboard
Add support for gRPC max connection age configuration
Description
Add support for server side gRPC max connection age configuration
References:
- Netty gRPC maxConnectionAge
- Netty gRPC maxConnectionAgeGrace
- Default gRPC KeepAlive configuration
Fixes #2999
Type of change
- [x] New feature (non-breaking change which adds functionality)
Feature/Issue validation/testing
- [ ] CI
- https://github.com/pytorch/serve/blob/master/test/pytest/test_gRPC_inference_api.py
- https://github.com/pytorch/serve/blob/master/test/pytest/test_gRPC_management_apis.py
- [ ] Manual test
# gRPC management connection timeout is set to 5 seconds
$ cat custom_config.properties
grpc_management_max_connection_age_ms=5000
grpc_management_max_connection_age_grace_ms=0
$ torchserve --ncs --start --model-store ./model_store --ts-config ./custom_config.properties
# Note: Model takes longer than 5 seconds to load
$ time python ts_scripts/torchserve_grpc_client.py register densenet161
## Check densenet161.mar in mar_set : set()
## Register marfile: https://torchserve.s3.amazonaws.com/mar_files/densenet161.mar
Failed to register model densenet161.
Socket closed
python ts_scripts/torchserve_grpc_client.py register densenet161 0.13s user 0.06s system 3% cpu 5.629 total
As can be seen from the test above, the gRPC management connection to register model is closed after the configured 5 seconds (max connection age).