bug: In the case of high concurrent requests, the default type of upstreams cannot achieve load balancing.

[pliwang]

Current Behavior

There are 30 routes corresponding to an upstream, including 5 nodes. When 3000 requests these 30 routes concurrently, the problem of 502 will occasionally occur.

Expected Behavior

After careful investigation, it is found that the default load balancing policy type of upstream can not reach the polling choice of nodes,Thereby causing the problem of 502.How should I avoid such a problem?

Error Logs

Steps to Reproduce

1. 30 routes
2. 30 routes correspond to an upstream.
3. Concurrent access
4. Check whether the number of requests processed by upstream services is balanced.

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[pliwang]

Error Logs

[Revolyssup]

Can you provide the upstream configuration? One of the route configuration as well maybe? And share the error log of APISIX when 502 occurs.

[pliwang]

The reason of 502 is that my service can only support about 1000 concurrency. I have five such services. When 3000 concurrency, I expect each service to be able to load balance to 600 to 1000 requests, but in fact, I find that the load of a service may be as high as more than 2000 requests. The following are the related configurations of and.

[pliwang]

{ "createdIndex": 62589, "modifiedIndex": 62589, "key": "/apisix/2-default-ns/routes/daadccdb", "value": { "name": "ingress_demo11_cloudspace_demo1_8001__/podip/h11", "desc": "Created by apisix-ingress-controller, DO NOT modify it manually", "labels": { "managed-by": "apisix-ingress-controller", "svcPort": "8001", "projectId": "2", "sourceType": "ingress", "sourceName": "demo11", "path": "/podip/h11", "serviceName": "demo1", "namespace": "cloudspace" }, "id": "daadccdb", "create_time": 1692237845, "update_time": 1692237845, "priority": 0, "status": 1, "uris": [ "/podip/h11", "/podip/h11/*" ], "upstream_id": "bb223adb" } }, { "createdIndex": 62606, "modifiedIndex": 62606, "key": "/apisix/2-default-ns/routes/de7714e6", "value": { "name": "ingress_demo7_cloudspace_demo1_8001__/podip/h7", "desc": "Created by apisix-ingress-controller, DO NOT modify it manually", "labels": { "managed-by": "apisix-ingress-controller", "svcPort": "8001", "projectId": "2", "sourceType": "ingress", "sourceName": "demo7", "path": "/podip/h7", "serviceName": "demo1", "namespace": "cloudspace" }, "id": "de7714e6", "create_time": 1692237848, "update_time": 1692237848, "priority": 0, "status": 1, "uris": [ "/podip/h7", "/podip/h7/*" ], "upstream_id": "bb223adb" } }

[pliwang]

{ "nodes": [ { "createdIndex": 62549, "modifiedIndex": 62632, "key": "/apisix/2-default-ns/upstreams/bb223adb", "value": { "name": "cloudspace_demo1_8001", "desc": "Created by apisix-ingress-controller, DO NOT modify it manually", "nodes": [ { "port": 8000, "host": "192.168.167.100", "weight": 100, "priority": 0 }, { "port": 8000, "host": "192.168.228.235", "weight": 100, "priority": 0 }, { "port": 8000, "host": "192.168.23.198", "weight": 100, "priority": 0 }, { "port": 8000, "host": "192.168.235.237", "weight": 100, "priority": 0 }, { "port": 8000, "host": "192.168.71.143", "weight": 100, "priority": 0 } ], "hash_on": "vars", "labels": { "managed-by": "apisix-ingress-controller", "svc": "demo1", "port": "8001", "namespace": "cloudspace" }, "id": "bb223adb", "create_time": 1692237843, "type": "roundrobin", "scheme": "http", "pass_host": "pass", "update_time": 1692599826 } } ], "dir": true, "key": "/apisix/2-default-ns/upstreams" }

[shreemaan-abhishek]

30 routes correspond to an upstream.

Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?

[Revolyssup]

@pliwang Can you provide the ApisixUpstream config or Ingress config, whatever you used with ingress controller to configure the upstream?

[pliwang]

30 routes correspond to an upstream.

Do you mean to say that each route corresponds to a certain upstream or each route corresponds to the same upstream?

Each route used corresponds to an upstream.

[pliwang]

@pliwang Can you provide the ApisixUpstream config or Ingress config, whatever you used with ingress controller to configure the upstream?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: apisix
  creationTimestamp: "2023-08-17T02:04:02Z"
  generation: 1
  labels:
    cube.gateway.apisix.projectId: "2"
  managedFields:
  - apiVersion: networking.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:rules: {}
    manager: OpenAPI-Generator
    operation: Update
    time: "2023-08-17T02:04:02Z"
  name: demo10
  namespace: cloudspace
  resourceVersion: "55774113"
  selfLink: /apis/extensions/v1beta1/namespaces/cloudspace/ingresses/demo10
  uid: e2f79de3-8282-4ac9-b55a-c5aafc01e979
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: demo1
          servicePort: 8001
        path: /podip/h10
        pathType: Prefix
status:
  loadBalancer: {}

There are 30 such ingress, and the other configurations are the same except for different paths.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.