bug: add_target when create healthchecker consume too much cpu

[dream110fly]

Current Behavior

we use apisix 2.13.1，api7-lua-resty-healthcheck package version is 3.2.0. we create one upstream with 1000 k8s pods，then update upstream with 1 second period. we find that one or more nginx process cpu load not stable, sometime cpu load reach 100%, then come down.

from lua stack flame, we can see that apisix locking_target_list when add_target of upstream one by one consume too much cpu

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1.create one upstream with 1000 pods 2.update upstream with 1 second period 3.watch nginx prcess cpu load

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

then update upstream with 1 second period

What does this mean?

create one upstream with 1000 pods

Is this a real use case environment?

[dream110fly]

call upstream admin api to update upstream periodicall， and the period is 1 second.

it is our real environment.

then update upstream with 1 second period

What does this mean?

create one upstream with 1000 pods

Is this a real use case environment?

[shreemaan-abhishek]

Interesting. I will take a look when I am free. Thanks for the report <3

Please send the flamegraph and other relevant files to my email [email protected]

[dream110fly]

https://github.com/Kong/kong/issues/7654 kong also have the same problem