Additional Events for RateLimitOptions

[pratap-bhaskar]

Background and Motivation

We have memory intensive application hosted in Kubernetes, this application's memory is directly proportional to the number of requests it can handle. We were using Semaphore to limit the concurrency and create internal queues for processing.

Now that .Net7 has a ConcurrencyLimiter we started investigating on the same, the idea (although non-proven) was to turn off the Readiness probe once the concurrency limit had reached so that the traffic wouldn't flow to that instance, once the limit is replenished we could turn on the Readiness probe to start allowing the traffic back to the pod.

Unfortunately, the RateLimiter option has only one event, and that occurs on the rejection of the request since the limit was breached.

public Func<OnRejectedContext, CancellationToken, ValueTask>? OnRejected { get; set; }

Even if we consume this event to turn off the probe, there is no way to bring the readiness probe back as there is no apparent way to check if the limits are replenished.

builder.Services.AddRateLimiter(_ =>
    {
        _.AddConcurrencyLimiter(policyName: "LimiterPolicy", options =>
        {
            options.PermitLimit = 10;
            // Only allow 10 request every 10 seconds
            options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
            // Only queue 3 requests when we go over that limit
            options.QueueLimit = 3;
        });
        _.OnRejected = async (context, token) =>
        {
            context.HttpContext.Response.StatusCode = 429;
            ConcurrencyLimitHolder.HasLimitReached = true;
        };
    });

///in the probe controller
[HttpGet("ready")]
    public IActionResult GetReadiness()
    {
        if (ConcurrencyLimitHolder.HasLimitReached)
            return BadRequest();
        
        return Ok();
    }

This isn't very useful, because the instance has already started rejecting the requests and the clients would have to retry, but the chances are that it would end up in the same instance.

Proposed API

An event triggered as soon as the rate limit is replenished/reset

public sealed class RateLimiterOptions
{
     public Func<OnReplinshedContext, CancellationToken, ValueTask>? OnReplinished { get; set; }
}

Usage Examples

using the same example as above

builder.Services.AddRateLimiter(_ =>
    {
        _.AddConcurrencyLimiter(policyName: "LimiterPolicy", options =>
        {
            options.PermitLimit = 10;
            // Only allow 10 request every 10 seconds
            options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
            // Only queue 3 requests when we go over that limit
            options.QueueLimit = 3;
        });
        _.OnRejected = async (context, token) =>
        {
            context.HttpContext.Response.StatusCode = 429;
            ConcurrencyLimitHolder.HasLimitReached = true;
        };
        _.OnReplenished = async(context,token) =>
       {
            ConcurrencyLimitHolder.HasLimitReached = false;
       }
    });

///in the probe controller
[HttpGet("ready")]
    public IActionResult GetReadiness()
    {
        if (ConcurrencyLimitHolder.HasLimitReached)
            return BadRequest();
        
        return Ok();
    }

[adityamandaleeka]

@BrennanConroy PTAL.