apisix icon indicating copy to clipboard operation
apisix copied to clipboard

fix(limit-req): Make Redis path atomic via EVAL + use hash key with TTL

Open falvaradorodriguez opened this issue 5 months ago • 5 comments

Description

The current limit-req Redis implementation uses two separate keys (excess and last) and updates them with multiple GET/SET operations.

  • Under concurrent load, this leads to race conditions:
  1. Several workers may read stale values in parallel and overwrite each other.
  2. As a result, the plugin allows more requests than expected, effectively bypassing the intended rate limit.
  • On Redis Cluster, the current approach is also problematic: atomic EVAL cannot be executed across two different keys located on different slots.

Solution

  • Store both values (excess and last) under a single Redis hash key, so the state is managed as one unit.

  • Use a single EVAL script that performs read → compute → write atomically inside Redis, removing race conditions. This approach is consistent with how the limit-count plugin already works.

  • Add a TTL to the key to avoid buildup of stale state.

  • Preserve existing semantics: the first request with no prior state does not consume tokens.

Which issue(s) this PR fixes:

Fixes #12592

Checklist

  • [x] I have explained the need for this PR and the problem it solves
  • [x] I have explained the changes or the new features added to this PR
  • [x] I have added tests corresponding to this change
  • [ ] I have updated the documentation to reflect this change
  • [x] I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

falvaradorodriguez avatar Sep 09 '25 15:09 falvaradorodriguez

‌Hi @falvaradorodriguez, we need to add the test case for this fix.

Baoyuantop avatar Sep 10 '25 09:09 Baoyuantop

Hi @falvaradorodriguez, there are failed CI that need fixing.

Baoyuantop avatar Sep 22 '25 08:09 Baoyuantop

Hi @falvaradorodriguez, there are failed CI that need fixing.

Hi @Baoyuantop!

The failures are not related to the current changes. Tests of the modified plugin seems to be fine.

Please, Can you review?

Thanks

[15:23:12] t/plugin/limit-req-redis-cluster.t ......... ok    15865 ms ( 0.01 usr  0.01 sys +  0.91 cusr  2.03 csys =  2.96 CPU)
[15:23:29] t/plugin/limit-req-redis.t ................. ok    16682 ms ( 0.02 usr  0.00 sys +  0.95 cusr  2.23 csys =  3.20 CPU)
2025/09/22 15:23:37 Processed 0 requests
[15:23:41] t/plugin/limit-req.t ....................... ok    11856 ms ( 0.02 usr  0.00 sys +  0.83 cusr  1.78 csys =  2.63 CPU)
[15:23:46] t/plugin/limit-req2.t ...................... ok     5010 ms ( 0.01 usr  0.00 sys +  0.81 cusr  0.42 csys =  1.24 CPU)
[15:23:47] t/plugin/limit-req3.t ...................... ok     1290 ms ( 0.00 usr  0.00 sys +  0.39 cusr  0.22 csys =  0.61 CPU)

falvaradorodriguez avatar Sep 22 '25 17:09 falvaradorodriguez

When will this PR be merged? I am having the same issue that this PR fixes 🙏

luarx avatar Oct 08 '25 13:10 luarx

I will promptly urge other community maintainers to review.

Baoyuantop avatar Oct 11 '25 03:10 Baoyuantop