multiple dotnet-monitors in connect mode in a machine will cause the process's diagnostic port not responsive

[LaurenceChau]

Description

dotnet-monitor can only expose one process's metrics, when we need to monitor multiple dotnet process metrics in a machine, we need to start multiple dotnet monitors.

The problem is that each dotnet-monitor in connect mode, it will try to discover all dotnet processes in the machine than expose metrics of the the target process . The process discovery step make no impact to the process diagnostic-port when there are little dotnet monitors running. However, when we run more than 20 dotnet-monitor in a machine, the diagnostic-port of the process will be come not responsive. And the not responsive problem of diagnostic-port of the process will persist even after we stopped all dotnet monitor.

Configuration

shared config

{
    "$schema": "https://aka.ms/dotnet-monitor-schema",
    "DiagnosticPort": {
        "ConnectionMode": "Connect"
    },
    "GlobalCounter": {
        "IntervalSeconds": 5
    },
    "Logging": {
        "Console": {
          "FormatterName": "Simple",
          "FormatterOptions": {
            "IncludeScopes": true,
            "TimestampFormat": "HH:mm:ss "
          }
        },
        "EventLog": {
          "LogLevel": {
            "Default": "Information",
            "Microsoft": "Warning",
            "Microsoft.Diagnostics": "Information",
            "Microsoft.Hosting.Lifetime": "Information"
          }
        },
        "LogLevel": {
          "Default": "Information",
          "Microsoft": "Warning",
          "Microsoft.Diagnostics": "Information",
          "Microsoft.Hosting.Lifetime": "Information"
        }
      }
    }
}

user specified config for each process

• The App.CloudService is a sample app of the dotnet/samples repo in the path core/workers/cloud-service. dotnet/sample
• we have 20 processes running. Port, metricPort and processAlias are different for each process.

{
    "urls": "https://*:<port>",
    "Metrics": {
        "Enabled": true,
        "Endpoints": "https://*:<metricPort>",
        "IncludeDefaultProviders": true,
        "MetricCount": 3
    },
    "DefaultProcess": {
        "Filters": [{
            "Key": "ProcessName",
            "Value": "App.CloudService",
            "MatchType": "Exact"
        },{
            "Key": "CommandLine",
            "Value": "<processAlias>",
            "MatchType": "Contains"
        }]
    }
}

Regression?

Other information

<B>Step to reproduce the problem. </B>

using the config above, run the command 20 times

# shared config are located in C:\ProgramData\dotnet-monitor 
dotnet monitor collect --configuration-file-path <dotnet-monitor1>\settings.json
dotnet monitor collect --configuration-file-path <dotnet-monitor2>\settings.json
...

Then you will start to see the warning: Cancelled waiting for diagnostic response from runtime in process xxxx

Although we can use Listen mode then we may need to pass the DOTNET_DiagnosticPorts=my_diag_port1 variable to the dotnet runtime however that may impact the way we manage our app.

It this is a bug, it worth fixing it.

Or if it is possible to control dotnet-monitor not try to discover all processes everytime when we have applied the process filter config?

[wiktork]

Possibly related https://github.com/dotnet/diagnostics/issues/3479