go-zero icon indicating copy to clipboard operation
go-zero copied to clipboard

ZRpc + Etcd + Docker Env, After the ZRpc client container slept for 1 hour, An ZRpc client request was initiated again, this request will exception and could not be recovered. The error "context deadline exceeded" appeared every 4 seconds in the log

Open ysw-jingzhe opened this issue 1 year ago • 4 comments

Describe the bug Zrpc + Etcd + Docker 容器微服务架构, Zrpc Client 端容器在刚启动时 rpc 之间请求正常,可以正常的通过ETCD发现服务端并请求到服务端数据。 但是在该Client 端容器睡眠1小时(1个小时内无任何外部请求) 后,再次触发Rpc客户端请求, 此时Rpc请求出现超时异常,且日志中每隔4秒出现一次 “error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240”。 此时检查 Docker 容器正常,容器中网络也正常。

备注:在不使用容器直接在服务器上运行该Client端的Go程序,就不会存在该问题; 在相同容器中使用 kratos 配套google.golang.org/grpc v1.46.2 不会存在该问题;

Zrpc+Etcd+Docker microservice Env, the Zrpc client container requests normally between RPC at the beginning of startup, and can discover the server and request server data through ETCD normally. However, after the client-side container had slept for 1 hour (with no external requests within the hour), the Rpc client request was triggered again. At this time, the Rpc request encountered a timeout exception and appeared every 4 seconds in the log "error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240".

Note: If the Go program on the client side is run directly on the server without using a container, this issue will not exist; Using Kratos + “google.golang.org/grpc v1.46.2” in the same container will not result in this issue;

Docker Image: alpine:latest or ubuntu 20.4 Etcd: etcd:v3.5.12 Zrpc: github.com/zeromicro/go-zero v1.7.2 Grpc: google.golang.org/grpc v1.67.0

To Reproduce Steps to reproduce the behavior, if applicable:

  1. The code is

    // client init
    conn = zrpc.MustNewClient(zrpc.RpcClientConf{
      Etcd: discov.EtcdConf{
         Hosts: ["127.0.0.1:2379"],
         Key: "examination.rpc"
      },
    })
    
    // client every request
    client := greet.NewGreetClient(conn.Conn())
    resp,err := client.Debug(ctx, in, opts...)
    
    
  2. The error is

    2024-10-11T09:33:40.108Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:44.109Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:48.111Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:52.113Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:56.115Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    
    

Expected behavior RPC requests are normal and stable

Screenshots

Environments (please complete the following information):

  • OS: Ubuntu
  • go-zero version v1.7.2
  • goctl version v.1.22.6
  • Docker Image: alpine:latest or ubuntu 20.4
  • Etcd: etcd:v3.5.12
  • Grpc: google.golang.org/grpc v1.67.0

More description

ysw-jingzhe avatar Oct 13 '24 04:10 ysw-jingzhe

I'll try to reproduce it.

zhoushuguang avatar Oct 14 '24 07:10 zhoushuguang

thanks

ysw-jingzhe avatar Oct 15 '24 03:10 ysw-jingzhe

Is there any progress? How to be stable? rpc for docker containers

ysw-jingzhe avatar Oct 24 '24 03:10 ysw-jingzhe

解决了吗

xuwuruoshui avatar Jun 09 '25 04:06 xuwuruoshui