CosmosDB with managed Identity - Provided AAD token has been expired

[pvpetrac]

• Operating system: Windows (app service and function apps both premium)
• [ ] nodejs
  • version: >=16

We are using libraries: "@azure/cosmos": "^3.16.3", "@azure/identity": "^2.1.0"

Cosmos sdk does not refresh aad token. We have issues with both app service and function apps. {"code":"Unauthorized","message":"Request blocked by Auth cosmos-public-serverless-scus : Provided AAD token has been expired since [7/17/2022 5:09:09 PM (UTC)]. Current server time is [7/18/2022 12:04:25 AM (UTC)]. Please refresh the AAD token.\r\nActivityId: xxxxx-xxxx-xxxxx-xxxx-xxxxxxxx, Microsoft.Azure.Documents.Common/2.14.0"}

Steps to reproduce the behavior:

1. this.client = new CosmosClient({endpoint: config.database.endpoint, aadCredentials: new ManagedIdentityCredential()})
2. Wait for some time
3. Initiate a new request

Expected behavior The documentation says that token should be refreshed by the sdk itself in interval.

Screenshots Once token expires, huge amount of issues starting to pop off.

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Cosmos:0.55361664,Azure.Identity:0.3827961,AppAuthentication:0.016587596'

[sajeetharan]

@pvpetrac Thanks for reporting the issue. We will investigate and get back on this!

[pvpetrac]

Meanwhile, we reverted to cosmos connection via key. FYI, couple of weeks ago, we have implemented Managed Identity for Service Bus and Storage Account, and there were no similar issues there.

[ssidnal]

Is there any workaround on this issue? It is effecting us on many environemnts including production.

[jay-most]

there is no workaround. The ADD token refresh feature is not implemented in NODEJS SDK. You will have to manually refresh.

[pvpetrac]

Can you please :pray: :pray: post some example? If I start diggin in, I'll have to invest time into it. I haven't found the way to access token from CosmosClient object. Maybe ManagedIdentityCredential should handle it? By documentation GetToken method is getting a new token, but how to reach existing one?

I appreciate your time @jay-most I hope you'll have the understanding for my client budget

[ssidnal]

there is no workaround. The ADD token refresh feature is not implemented in NODEJS SDK. You will have to manually refresh.

@jay-most We have been using cosmos connection using System Managed Identity new CosmosClient({endpoint, aadCredentials: new DefaultAzureCredential()}) from last 5 months in all our environments. Never seen this issue, If refresh was not implemented how was this working all these days and started breaking now in the last 2 weeks? I feel something is not right here.

[sajeetharan]

@ssidnal @pvpetrac We'll prioritise this. Can you provide a sample with your code that would help

[ssidnal]

Thanks, @sajeetharan for confirming. We have been creating instances of the database like below,

import {Container, CosmosClient, Database} from '@azure/cosmos';
import {DefaultAzureCredential} from '@azure/identity';

const endpoint = process.env['ENDPOINT'];
const dbName = process.env['DB_NAME'];

const client = new CosmosClient({endpoint, aadCredentials: new DefaultAzureCredential()});
const database = client.database(dbName);

export const container =database.container('container_name') ;

[Delapouite]

@ssidnal Could it possibly be related to https://github.com/Azure/azure-sdk-for-js/issues/22722 ?

In other words, did you upgrade @azure/identity recently to 2.1.0?

[ssidnal]

@ssidnal Could it possibly be related to #22722 ?

In other words, did you upgrade @azure/identity recently to 2.1.0?

Thanks @Delapouite. Yes we are using 2.1.0 version of @azure/identity. Let us verify by downgrading it to 2.0.5. Thank you for the suggestion.

[sajeetharan]

@Delapouite @ssidnal Looks like similar issue https://github.com/Azure/azure-sdk-for-js/issues/22722#issuecomment-1205536176 , we are checking on it and will provide an update soon!

[BenLegra]

Hi, as mentioned in this issue #22722, there's something wrong in @azure/identity library, in appServiceMsiApp2019.ts at this line. The expires_on value (in payload returned by the local /msi/token endpoint) is a stringified value of a Unix Epoch instead of a date.

This line: return Date.parse(requestBody.expires_on! as string); should be replaced by: return parseInt(requestBody.expires_on!) * 1000;

I've tested this by manually editing the file in node_modules and ran that for 3 days in an App Service: the tokens are correctly updated and so, accepted by services such as CosmosDb, Keyvaults, etc.

[ghost]

Do we have any updates on this issue and a fix in the SDK?

[jeremymeng]

@asdaandrewhaigh a fix has been merged in PR #23232. It should be available in the next version of identity soon.

[jeremymeng]

@BenLegra @asdaandrewhaigh @azure/identity 3.0.0 has been released, which includes fix to the expires_on parsing issue.

[sajeetharan]

This is completed and released!

[tlmae10]

Hey all, after upgrading to: "@azure/cosmos": "^3.17.2", "@azure/identity": "^3.1.1"

we still experienced the problem with expired AAD tokens within a managed identity. Is anyone else having the same problem?