http 503 exception occurs during Pressure test and continuous reload kong.yml via Admin API (db_less)

[kanbo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

kong 2.8.1

Current Behavior

1. When I did a pressure test on Kong via jmeter(60 threads)，During this process, I request the admin API /config to reload kong.yml via jmeter(1 thread), http 503 Service Temporarily Unavailable happens during Pressure test(The probability is 0.29%)

2. if I do not reload kong.yml During pressure test ,there was not http 503 error

Expected Behavior

When I did a pressure test on Kong，all request return http 200 , have no http 503 exception In the production environment, we will frequently update kong.yml

Steps To Reproduce

1、kong version 2.8.1, CentOS Linux release 7.3.1611 (Core) 8C
2、yum install kong-2.8.1.el7.amd64.rpm
3、update kong.conf and start kong
4、begin  pressure test on Kong  and reload kong.yml via jmeter

Anything else?

1. kong.config worker_consistency = strict log_level = info mem_cache_size = 256m database = off

2. kong.yml _format_version: "2.1"

_transform: true

services:

• name: static-front-perf-sd host: static-front-perf-sd connect_timeout: 10000 protocol: http routes:
  • strip_path: false preserve_host: true name: static-front-perf-sd hosts:
    • static.test.com methods:
    • GET
    • POST paths:
    • /
• name: testin-web-perf-sd host: testin-web-perf-sd connect_timeout: 10000 protocol: http routes:
  • strip_path: false preserve_host: true name: testin-web-perf-sd hosts:
    • in.test.com methods:
    • GET
    • POST paths:
    • / upstreams:
• name: testin-web-perf-sd healthchecks: active: https_verify_certificate: true type: http http_path: /detect timeout: 3 concurrency: 10 healthy: successes: 3 interval: 10 unhealthy: tcp_failures: 3 timeouts: 3 http_failures: 3 interval: 10 passive: type: http healthy: successes: 3 unhealthy: tcp_failures: 3 timeouts: 3 http_failures: 3 targets:
  • target: 10.16.95.67:10829 weight: 10
  • target: 10.16.95.68:10829 weight: 10
• name: testin-web-perf-sd-grayroute healthchecks: active: https_verify_certificate: true type: http http_path: /detect timeout: 3 concurrency: 10 healthy: successes: 3 interval: 10 unhealthy: tcp_failures: 3 timeouts: 3 http_failures: 3 interval: 10 passive: type: http healthy: successes: 3 unhealthy: tcp_failures: 3 timeouts: 3 http_failures: 3 targets:
  • target: 10.16.95.69:10829 weight: 5
  • target: 10.16.95.67:10829 weight: 5
  • target: 10.16.95.68:10829 weight: 5

3. there was no error log in error.log

4. try to add debug code in kong why appear http 503，because balancer.healthy is false, kong/runloop/balancer/init.lua --line 289；kong/runloop/balancer/balancers.lua --line 563 ， so I add some debug log in balancers.lua and targets.lua balancers.lua

targets.lua

Continue pressure test the error.log when execute method create_balancer_exclusive() , Sometimes the following occurs ：target.addresses is has value and address.available is true，so it never update balancer.healthy During create_balancer in error.log ,some nginx worker use pre cache， is there a bug in kong cache?

5. how can I solve or avoid the problem use different upstream name when request admin api /config? when targets.fetch_targets(upstream) in balancers.lua do not use cache , read from db can solve this problem(db_less mode)

6. similar issue https://github.com/Kong/kong/issues/7543

[kanbo]

@bungle

[esatterwhite]

Sounds related to #8976

[raoxiaoyan]

@kanbo Have you solved the problem?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team