Allow configuration for request limit

[LaurenceChau]

Background and Motivation

I was querying the /livemetrics endpoint to get metrics of all running dotnet app. However when I sent 4 http requests to the endpoint at the same time it reached the dotnet-monitor limit.

This was the log I found in the dotnet-monitor.

{
    "Timestamp": "2023-04-28T09:03:10.0074525Z",
    "EventId": 1,
    "LogLevel": "Error",
    "Category": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController",
    "Message": "Request failed.",
    "Exception": "Microsoft.Diagnostics.Monitoring.WebApi.TooManyRequestsException: Rate limit reached.    at Microsoft.Diagnostics.Monitoring.WebApi.EgressOperationStore.AddOperation(IEgressOperation egressOperation, String limitKey)    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.RegisterOperation(IEgressOperation egressOperation, String limitKey)    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.RegisterCurrentHttpResponseAsOperation(IProcessInfo processInfo, String artifactType, String tags, IArtifactOperation operation)    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.Result(String artifactType, String providerName, IArtifactOperation operation, IProcessInfo processInfo, String tags, Boolean asAttachment)    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass38_0.\u003C\u003CInvokeForProcess\u003Eb__0\u003Ed.MoveNext() --- End of stack trace from previous location ---    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass40_0\u00601.\u003C\u003CInvokeForProcess\u003Eb__0\u003Ed.MoveNext() --- End of stack trace from previous location ---    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagControllerExtensions.InvokeService[T](ControllerBase controller, Func\u00601 serviceCall, ILogger logger)",
    "State": {
        "Message": "Request failed.",
        "{OriginalFormat}": "Request failed."
    },
    "Scopes": [
        {
            "Message": "SpanId:1d8f0c8172e33e99, TraceId:4f2acf6e2414db5c7b3bf0eaa8a91606, ParentId:0000000000000000",
            "SpanId": "1d8f0c8172e33e99",
            "TraceId": "4f2acf6e2414db5c7b3bf0eaa8a91606",
            "ParentId": "0000000000000000"
        },
        {
            "Message": "ConnectionId:0HMQ71KHO6FI4",
            "ConnectionId": "0HMQ71KHO6FI4"
        },
        {
            "Message": "RequestPath:/livemetrics RequestId:0HMQ71KHO6FI4:00000001",
            "RequestId": "0HMQ71KHO6FI4:00000001",
            "RequestPath": "/livemetrics"
        },
        {
            "Message": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.CaptureMetrics (Microsoft.Diagnostics.Monitoring.WebApi)",
            "ActionId": "e0a84220-5fe0-4a6b-9ddd-1a9d7df14e99",
            "ActionName": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.CaptureMetrics (Microsoft.Diagnostics.Monitoring.WebApi)"
        },
        {
            "Message": "ArtifactType:livemetrics",
            "ArtifactType": "livemetrics"
        }
    ]
}

{
    "Timestamp": "2023-04-28T09:11:00.0080559Z",
    "EventId": 6,
    "LogLevel": "Warning",
    "Category": "Microsoft.Diagnostics.Monitoring.WebApi.RequestLimitTracker",
    "Message": "Request limit for endpoint reached. Limit: 3, oustanding requests: 4",
    "State": {
        "Message": "Request limit for endpoint reached. Limit: 3, oustanding requests: 4",
        "limit": 3,
        "requests": 4,
        "{OriginalFormat}": "Request limit for endpoint reached. Limit: {limit}, oustanding requests: {requests}"
    },
    "Scopes": [
        {
            "Message": "SpanId:29b29f057694007b, TraceId:033ae614b446d68525a1bf0de164c368, ParentId:0000000000000000",
            "SpanId": "29b29f057694007b",
            "TraceId": "033ae614b446d68525a1bf0de164c368",
            "ParentId": "0000000000000000"
        },
        {
            "Message": "ConnectionId:0HMQ71KHO6FTR",
            "ConnectionId": "0HMQ71KHO6FTR"
        },
        {
            "Message": "RequestPath:/livemetrics RequestId:0HMQ71KHO6FTR:00000001",
            "RequestId": "0HMQ71KHO6FTR:00000001",
            "RequestPath": "/livemetrics"
        },
        {
            "Message": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.CaptureMetrics (Microsoft.Diagnostics.Monitoring.WebApi)",
            "ActionId": "e0a84220-5fe0-4a6b-9ddd-1a9d7df14e99",
            "ActionName": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.CaptureMetrics (Microsoft.Diagnostics.Monitoring.WebApi)"
        },
        {
            "Message": "ArtifactType:livemetrics",
            "ArtifactType": "livemetrics"
        },
        {
            "Message": "ArtifactSource_ProcessId:77091 ArtifactSource_RuntimeInstanceCookie:00dace6bc4c84d1c83f6ae685a5ce990",
            "ArtifactSource_ProcessId": "77091",
            "ArtifactSource_RuntimeInstanceCookie": "00dace6bc4c84d1c83f6ae685a5ce990"
        }
    ]
}

Try to find the limit value from the source code and it turn out that the request limit value is hard code within the code https://github.com/dotnet/dotnet-monitor/blob/main/src/Microsoft.Diagnostics.Monitoring.WebApi/RequestThrottling/RequestLimitTracker.cs#L28-L63

        public RequestLimitTracker(ILogger<RequestLimitTracker> logger)
        {
            //CONSIDER Should we have configuration for these?

            _requestLimitTable.Add(Utilities.ArtifactType_Dump, 1);
            _requestLimitTable.Add(Utilities.ArtifactType_GCDump, 1);
            _requestLimitTable.Add(Utilities.ArtifactType_Logs, 3);
            _requestLimitTable.Add(Utilities.ArtifactType_Trace, 3);
            _requestLimitTable.Add(Utilities.ArtifactType_Metrics, 3);
            _requestLimitTable.Add(Utilities.ArtifactType_Stacks, 1);

            _logger = logger;
        }

Proposed Feature

Proposal to allow configure the hard coded livemetrics request limit.

Usage Examples

This can allow sending over 3 http requests to the /livemetrics endpoint at the same time in order to scrape multiple dotnet app livemetrics .

[wiktork]

Another possible approach would be to enforce these limits on a per process basis rather than globally. I think this is a reasonable feature. Some considerations:

• Certain operations (dumps, stacks) should strictly be 1 per process
• There are runtime limitations to the amount of event pipe sessions per process, regardless of the limits we have here.