Monitor REST Endpoints with Microsoft Managed Identity auth endpoint

[delema98]

I want to utilize the http check functionality to monitor REST endpoints. For authentication OAuth is used in combination with the standard Microsoft managed identity.

With the powershell command below, I am able to retrieve a token and then afterwards call the heartbeat endpoint:

Token Call:

$response = Invoke-WebRequest -Uri http://localhost:4567/api/token -Headers @{ Metadata = "true" } -UseBasicParsing
ManagedIdentityContent = $response.Content | ConvertFrom-Json
$token = $ManagedIdentityContent.access_token

Heartbeat Call:

$headers = @{"Authorization" = "Bearer $token"}
Invoke-WebRequest -Uri http://localhost:4567/api/heartbeat -Method Post -Headers $headers -UseBasicParsing

When utilizing the datadog agent with the configuration (In general client id and client secret is not needed):

init_config:

instances:
  - name: HeartbeatEndpointCheck
    url: 'http://localhost:4567/api/heartbeat'
    method: post
   
    auth_token:
      reader:
        type: oauth
        url: 'http://localhost:4567/api/token'
        client_id: test
        client_secret: test
        options: 
          include_client_id: false
      writer:
        type: header
        name: Authorization
        value: 'Bearer <TOKEN>'

I get the these errors:

Output of the agent logs

2024-10-01 14:17:02 CEST | CORE | ERROR | (pkg/collector/worker/check_logger.go:71 in Error) | check:http_check | Error running check: [{"message": "(insecure_transport) OAuth 2 MUST utilize https.", "traceback": "Traceback (most recent call last):\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\checks\\base.py\", line 1228, in run\n self.check(instance)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\http_check\\http_check.py\", line 130, in check\n r = getattr(self.http, http_method)(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 355, in post\n return self._request('post', url, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 397, in _request\n self.handle_auth_token(method=method, url=url, default_options=self.options)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 545, in handle_auth_token\n self.auth_token_handler.poll(**request)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 748, in poll\n token = self.reader.read(**request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 847, in read\n response = oauth.fetch_token(**self._fetch_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\requests_oauthlib\\oauth2_session.py\", line 268, in fetch_token\n raise InsecureTransportError()\noauthlib.oauth2.rfc6749.errors.InsecureTransportError: (insecure_transport) OAuth 2 MUST utilize https.\n"}]
2024-10-01 14:17:02 CEST | CORE | ERROR | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | http_check:HeartbeatEndpointCheck:68be6039a67f068c | (http_check.py:184) | Unhandled exception (insecure_transport) OAuth 2 MUST utilize https.. Connection failed after 0 ms
2024-10-01 14:17:02 CEST | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:133 in LogMessage) | http_check:HeartbeatEndpointCheck:68be6039a67f068c | (http_check.py:119) | Connecting to http://localhost:4567/api/heartbeat
2024-10-01 14:17:02 CEST | CORE | DEBUG | (pkg/collector/python/check.go:99 in runCheckImpl) | Running python check http_check (version: '9.7.0', id: 'http_check:HeartbeatEndpointCheck:68be6039a67f068c')

I managed to avoid this error by setting the environment variable OAUTHLIB_INSECURE_TRANSPORT = 1

However I ran then into another issue:

2024-10-01 14:18:27 CEST | CORE | ERROR | (pkg/collector/worker/check_logger.go:71 in Error) | check:http_check | Error running check: [{"message": "'>=' not supported between instances of 'float' and 'NoneType'", "traceback": "Traceback (most recent call last):\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\checks\\base.py\", line 1228, in run\n self.check(instance)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\http_check\\http_check.py\", line 130, in check\n r = getattr(self.http, http_method)(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 355, in post\n return self._request('post', url, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 397, in _request\n self.handle_auth_token(method=method, url=url, default_options=self.options)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 545, in handle_auth_token\n self.auth_token_handler.poll(**request)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 748, in poll\n token = self.reader.read(**request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 836, in read\n if self._token is None or get_timestamp() >= self._expiration or 'error' in request:\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nTypeError: '>=' not supported between instances of 'float' and 'NoneType'\n"}]
2024-10-01 14:18:27 CEST | CORE | ERROR | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | http_check:HeartbeatEndpointCheck:68be6039a67f068c | (http_check.py:184) | Unhandled exception '>=' not supported between instances of 'float' and 'NoneType'. Connection failed after 0 ms
2024-10-01 14:18:27 CEST | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:133 in LogMessage) | http_check:HeartbeatEndpointCheck:68be6039a67f068c | (http_check.py:119) | Connecting to http://localhost:4567/api/heartbeat
2024-10-01 14:18:27 CEST | CORE | DEBUG | (pkg/collector/python/check.go:99 in runCheckImpl) | Running python check http_check (version: '9.7.0', id: 'http_check:HeartbeatEndpointCheck:68be6039a67f068c')

I then changed to configuration to directly call the real auth endpoint (localhost:4567/ acts as a reverse proxy). Endpoint to get token from (docs: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-use-vm-token) GET 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/' HTTP/1.1 Metadata: true

init_config:

instances:
  - name: HeartbeatEndpointCheck
    url: 'http://localhost:4567/api/heartbeat'
    method: post
   
    auth_token:
      reader:
        type: oauth
        url: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=api://xxx'
        client_id: test
        client_secret: test
        options: 
          include_client_id: false
      writer:
        type: header
        name: Authorization
        value: 'Bearer <TOKEN>'

and got the following output:

2024-10-01 14:22:39 CEST | CORE | ERROR | (pkg/collector/worker/check_logger.go:71 in Error) | check:http_check | Error running check: [{"message": "(invalid_request) Request Method not supported for the API", "traceback": "Traceback (most recent call last):\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\checks\\base.py\", line 1228, in run\n self.check(instance)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\http_check\\http_check.py\", line 130, in check\n r = getattr(self.http, http_method)(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 355, in post\n return self._request('post', url, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 397, in _request\n self.handle_auth_token(method=method, url=url, default_options=self.options)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 545, in handle_auth_token\n self.auth_token_handler.poll(**request)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 748, in poll\n token = self.reader.read(**request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\datadog_checks\\base\\utils\\http.py\", line 847, in read\n response = oauth.fetch_token(**self._fetch_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\requests_oauthlib\\oauth2_session.py\", line 406, in fetch_token\n self._client.parse_request_body_response(r.text, scope=self.scope)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\oauthlib\\oauth2\\rfc6749\\clients\\base.py\", line 427, in parse_request_body_response\n self.token = parse_token_response(body, scope=scope)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\oauthlib\\oauth2\\rfc6749\\parameters.py\", line 441, in parse_token_response\n validate_token_parameters(params)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\oauthlib\\oauth2\\rfc6749\\parameters.py\", line 448, in validate_token_parameters\n raise_from_error(params.get('error'), params)\n File \"C:\\Program Files\\Datadog\\Datadog Agent\\embedded3\\Lib\\site-packages\\oauthlib\\oauth2\\rfc6749\\errors.py\", line 399, in raise_from_error\n raise cls(**kwargs)\noauthlib.oauth2.rfc6749.errors.InvalidClientIdError: (invalid_request) Request Method not supported for the API\n"}]
2024-10-01 14:22:39 CEST | CORE | ERROR | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | http_check:HeartbeatEndpointCheck:76d1ac23da76ab6a | (http_check.py:184) | Unhandled exception (invalid_request) Request Method not supported for the API. Connection failed after 9 ms

Is there a way to firstly allow http calls without setting the experimental env variable and secondly, is it possible to configure the datadog agent in a way that it supports the managed identity endpoints from microsoft?