grpc-go icon indicating copy to clipboard operation
grpc-go copied to clipboard

Flaky test: EDS_ClusterResourceUpdates

Open easwars opened this issue 8 months ago • 1 comments

--- FAIL: Test (11.86s)
    --- FAIL: Test/EDS_ClusterResourceUpdates (5.01s)
        eds_impl_test.go:782: Created new snapshot cache...
        tlogger.go:116: INFO server.go:690 [core] [Server #640]Server created  (t=+4.064[235](https://github.com/grpc/grpc-go/actions/runs/13392143576/job/37402121862?pr=8103#step:8:236)ms)
        eds_impl_test.go:782: Registered Aggregated Discovery Service (ADS)...
        eds_impl_test.go:782: xDS management server serving at: 127.0.0.1:39351...
        tlogger.go:116: INFO server.go:886 [core] [Server #640 ListenSocket #641]ListenSocket created  (t=+4.343646ms)
        tlogger.go:116: INFO server.go:690 [core] [Server #642]Server created  (t=+5.87364ms)
        stubserver.go:300: Started test service backend at "127.0.0.1:32825"
        tlogger.go:116: INFO server.go:886 [core] [Server #642 ListenSocket #643]ListenSocket created  (t=+6.18882ms)
        tlogger.go:116: INFO server.go:690 [core] [Server #644]Server created  (t=+6.529487ms)
        stubserver.go:300: Started test service backend at "127.0.0.1:37725"
        server.go:229: Created new resource snapshot...
        server.go:235: Updated snapshot cache with resource snapshot...
        tlogger.go:116: INFO server.go:886 [core] [Server #644 ListenSocket #645]ListenSocket created  (t=+6.91156ms)
        tlogger.go:116: INFO pool.go:[242](https://github.com/grpc/grpc-go/actions/runs/13392143576/job/37402121862?pr=8103#step:8:243) [xds] xDS node ID: 19ef5c01-324f-4ebd-9a01-f1e2d10bd2b2  (t=+7.615399ms)
        tlogger.go:116: INFO clientconn.go:1679 [core] original dial target is: "whatever:///test.service"  (t=+7.784585ms)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #646]Channel created  (t=+7.859545ms)
        tlogger.go:116: INFO clientconn.go:207 [core] [Channel #646]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"whatever", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/test.service", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}  (t=+7.950756ms)
        tlogger.go:116: INFO clientconn.go:208 [core] [Channel #646]Channel authority set to "test.service"  (t=+8.034142ms)
        tlogger.go:116: INFO resolver_wrapper.go:211 [core] [Channel #646]Resolver state updated: {
              "Addresses": null,
              "Endpoints": [],
              "ServiceConfig": {
                "Config": {
                  "Config": null,
                  "Methods": {}
                },
                "Err": null
              },
              "Attributes": {
                "\u003c%!p(xdsclient.clientKeyType=grpc.xds.internal.client.Client)\u003e": "\u003c0xc000395160\u003e"
              }
            } (service config updated)  (t=+8.341016ms)
        tlogger.go:116: INFO balancer_wrapper.go:122 [core] [Channel #646]Channel switches to new LB policy "cds_experimental"  (t=+8.462123ms)
        tlogger.go:116: INFO cdsbalancer.go:114 [xds] [cds-lb 0xc000548180] Created  (t=+8.552702ms)
        tlogger.go:116: INFO cdsbalancer.go:126 [xds] [cds-lb 0xc000548180] xDS credentials in use: false  (t=+8.595522ms)
        tlogger.go:116: INFO cdsbalancer.go:301 [xds] [cds-lb 0xc000548180] Received balancer config update: {
              "LoadBalancingConfig": null,
              "Cluster": "cluster-my-service-client-side-xds"
            }  (t=+8.665684ms)
        tlogger.go:116: INFO clientconn.go:1679 [core] original dial target is: "passthrough:///127.0.0.1:39351"  (t=+8.799955ms)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #647]Channel created  (t=+8.868[243](https://github.com/grpc/grpc-go/actions/runs/13392143576/job/37402121862?pr=8103#step:8:244)ms)
        tlogger.go:116: INFO clientconn.go:207 [core] [Channel #647]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/127.0.0.1:39351", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}  (t=+8.959523ms)
        tlogger.go:116: INFO clientconn.go:208 [core] [Channel #647]Channel authority set to "127.0.0.1:39351"  (t=+9.044472ms)
        tlogger.go:116: INFO resolver_wrapper.go:211 [core] [Channel #647]Resolver state updated: {
              "Addresses": [
                {
                  "Addr": "127.0.0.1:39351",
                  "ServerName": "",
                  "Attributes": null,
                  "BalancerAttributes": null,
                  "Metadata": null
                }
              ],
              "Endpoints": [
                {
                  "Addresses": [
                    {
                      "Addr": "127.0.0.1:39351",
                      "ServerName": "",
                      "Attributes": null,
                      "BalancerAttributes": null,
                      "Metadata": null
                    }
                  ],
                  "Attributes": null
                }
              ],
              "ServiceConfig": null,
              "Attributes": null
            } (resolver returned new addresses)  (t=+9.374109ms)
        tlogger.go:116: INFO balancer_wrapper.go:122 [core] [Channel #647]Channel switches to new LB policy "pick_first"  (t=+9.552613ms)
        tlogger.go:116: INFO clientconn.go:857 [core] [Channel #647 SubChannel #648]Subchannel created  (t=+9.64757ms)
        tlogger.go:116: INFO clientconn.go:563 [core] [Channel #647]Channel Connectivity change to CONNECTING  (t=+9.719375ms)
        tlogger.go:116: INFO clientconn.go:1224 [core] [Channel #647 SubChannel #648]Subchannel Connectivity change to CONNECTING  (t=+9.848326ms)
        tlogger.go:116: INFO clientconn.go:1344 [core] [Channel #647 SubChannel #648]Subchannel picks a new address "127.0.0.1:39351" to connect  (t=+9.918868ms)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #647]Channel exiting idle mode  (t=+10.458447ms)
        tlogger.go:116: INFO clientconn.go:1224 [core] [Channel #647 SubChannel #648]Subchannel Connectivity change to READY  (t=+11.287438ms)
        tlogger.go:116: INFO clientconn.go:563 [core] [Channel #647]Channel Connectivity change to READY  (t=+11.506648ms)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #646]Channel exiting idle mode  (t=+11.58218ms)
        logging.go:30: nodeID "19ef5c01-324f-4ebd-9a01-f1e2d10bd2b2" requested type.googleapis.com/envoy.config.cluster.v3.Cluster[cluster-my-service-client-side-xds] and known map[]. Diff [cluster-my-service-client-side-xds]
        logging.go:30: respond type.googleapis.com/envoy.config.cluster.v3.Cluster[cluster-my-service-client-side-xds] version "" with version "1"
        tlogger.go:116: INFO cdsbalancer.go:425 [xds] [cds-lb 0xc000548180] Received Cluster resource: {
              "ClusterType": 0,
              "ClusterName": "cluster-my-service-client-side-xds",
              "EDSServiceName": "endpoints-my-service-client-side-xds",
              "LRSServerConfig": null,
              "SecurityCfg": null,
              "MaxRequests": null,
              "DNSHostName": "",
              "PrioritizedClusterNames": null,
              "LBPolicy": [
                {
                  "xds_wrr_locality_experimental": {
                    "childPolicy": [
                      {
                        "round_robin": {}
                      }
                    ]
                  }
                }
              ],
              "OutlierDetection": null,
              "Raw": {
                "type_url": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
                "value": "CiJjbHVzdGVyLW15LXNlcnZpY2UtY2xpZW50LXNpZGUteGRzGioKAhoAEiRlbmRwb2ludHMtbXktc2VydmljZS1jbGllbnQtc2lkZS14ZHMQAw=="
              },
              "TelemetryLabels": {
                "csm.service_name": "unknown",
                "csm.service_namespace_name": "unknown"
              }
            }  (t=+13.328757ms)
        tlogger.go:116: INFO clusterresolver.go:85 [xds] [xds-cluster-resolver-lb 0xc000616000] Created  (t=+13.420118ms)
        logging.go:30: nodeID "19ef5c01-324f-4ebd-9a01-f1e2d10bd2b2" requested type.googleapis.com/envoy.config.cluster.v3.Cluster[cluster-my-service-client-side-xds] and known map[cluster-my-service-client-side-xds:{}]. Diff []
        logging.go:30: open watch 1 for type.googleapis.com/envoy.config.cluster.v3.Cluster[cluster-my-service-client-side-xds] from nodeID "19ef5c01-324f-4ebd-9a01-f1e2d10bd2b2", version "1"
        tlogger.go:116: INFO cdsbalancer.go:466 [xds] [cds-lb 0xc000548180] Created child policy 0xc000616000 of type cluster_resolver_experimental  (t=+13.869549ms)
        eds_impl_test.go:834: EmptyCall() failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready
        tlogger.go:116: WARNING ads_stream.go:514 [xds] [xds-client 0xc00057eb40] [xds-channel 0xc0007408a0] [ads-stream 0xc00002ae00] ADS stream closed: rpc error: code = Canceled desc = context canceled  (t=+5.004317883s)
        tlogger.go:116: INFO ads_stream.go:178 [xds] [xds-client 0xc00057eb40] [xds-channel 0xc0007408a0] [ads-stream 0xc00002ae00] Stopping ADS stream  (t=+5.004387864s)
        tlogger.go:116: INFO clientconn.go:563 [core] [Channel #647]Channel Connectivity change to SHUTDOWN  (t=+5.004464698s)
        tlogger.go:116: INFO resolver_wrapper.go:111 [core] [Channel #647]Closing the name resolver  (t=+5.004516434s)
        tlogger.go:116: INFO balancer_wrapper.go:160 [core] [Channel #647]ccBalancerWrapper: closing  (t=+5.00457815s)
        tlogger.go:116: INFO clientconn.go:1224 [core] [Channel #647 SubChannel #648]Subchannel Connectivity change to SHUTDOWN  (t=+5.004694598s)
        tlogger.go:116: INFO clientconn.go:1552 [core] [Channel #647 SubChannel #648]Subchannel deleted  (t=+5.004757[245](https://github.com/grpc/grpc-go/actions/runs/13392143576/job/37402121862?pr=8103#step:8:246)s)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #647]Channel deleted  (t=+5.004948262s)
        tlogger.go:116: INFO channel.go:145 [xds] [xds-client 0xc00057eb40] [xds-channel 0xc0007408a0] Shutdown  (t=+5.005033061s)
        tlogger.go:116: INFO clientimpl.go:228 [xds] [xds-client 0xc00057eb40] Shutdown  (t=+5.00509126s)
        tlogger.go:116: INFO clientconn.go:563 [core] [Channel #646]Channel Connectivity change to SHUTDOWN  (t=+5.005146603s)
        tlogger.go:116: INFO resolver_wrapper.go:111 [core] [Channel #646]Closing the name resolver  (t=+5.005192059s)
        tlogger.go:116: INFO balancer_wrapper.go:160 [core] [Channel #646]ccBalancerWrapper: closing  (t=+5.00527843s)
        tlogger.go:116: INFO clusterresolver.go:355 [xds] [xds-cluster-resolver-lb 0xc000616000] Shutdown  (t=+5.005413883s)
        tlogger.go:116: INFO cdsbalancer.go:391 [xds] [cds-lb 0xc000548180] Shutdown  (t=+5.005479987s)
        tlogger.go:116: INFO clientconn.go:333 [core] [Channel #646]Channel deleted  (t=+5.005550338s)
        tlogger.go:116: INFO server.go:822 [core] [Server #642 ListenSocket #643]ListenSocket deleted  (t=+5.005672136s)
        tlogger.go:116: INFO server.go:822 [core] [Server #644 ListenSocket #645]ListenSocket deleted  (t=+5.005789326s)
        tlogger.go:116: INFO server.go:822 [core] [Server #640 ListenSocket #641]ListenSocket deleted  (t=+5.0058983s)
FAIL
FAIL	google.golang.org/grpc/xds/internal/balancer/clusterresolver/e2e_test	21.169s

easwars avatar Feb 19 '25 18:02 easwars

https://github.com/grpc/grpc-go/actions/runs/13392143576/job/37402121862?pr=8103

easwars avatar Feb 19 '25 18:02 easwars

Ran a million times without a failure on Forge.

easwars avatar Nov 11 '25 02:11 easwars