help request: performance with blacklist plugins

[Lewisyixin]

Description

When I use the ip-restriction plugin and configure 2000 IP lists at the same time, the TP99 of apisix will increase significantly. Is this because apisix will be blocked when loading too much etcd data? Is there any good way to solve this problem?

Environment

• APISIX version (run apisix version): 3.5
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.21.4.2

[Lewisyixin]

[Lewisyixin]

I've done some more testing and found that any time a change to etcd is pushed to the apisix node, TP99 fluctuates. For example, modifying the description field of a routing rule at 5000qps can also cause latency fluctuations. A 32C CPU on the node is sufficient. Are there any better optimization solutions?

By the way, our TP99 statistics are collected on the client side, as minor congestion within apisix may not be captured by the TP99 statistics of apisix itself.

[Baoyuantop]

Can you test it with a newer version? The current version is too old.

[Lewisyixin]

@Baoyuantop I have test with 3.13 and reached the same conclusion. I used one apisix node with 16Core and one etcd node.

Initially I only had one routing configuration and one upstream configuration. Then I started stress testing. I inserted 2000 routing rules using a shell for loop. At this time, the fluctuation of the first red box in the figure appears.

Next, I selected a route with no traffic and enabled and disabled continuous clicks. At this time, the second red box fluctuation appeared.

[Lewisyixin]

This means that when the traffic is large, the action of apisix monitoring etcd configuration and sending it to the memory in sequence has a great impact on the performance of the link.

[Baoyuantop]

Hi @Lewisyixin, to clarify, the latency fluctuations were triggered by APISIX configuration changes, not by an increase in the number of IP lists?

[Lewisyixin]

HI @Baoyuantop , yes, that's my verification result.