MongoDB AWS auth rotating credentials support

[vkarpov15]

Discussed in https://github.com/Automattic/mongoose/discussions/12698

^{Originally posted by MrDoowie November 17, 2022} We're trying to use the MongoDB AWS authentication system for securing our connection (https://github.com/mongodb/specifications/blob/master/source/auth/auth.rst#mongodb-aws). The credentials we provide to it are retrieved through the '@aws-sdk/client-sts' package (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-sts/index.html).

Our initial connection is working fine, but we're trying to work out how to refresh these credentials before they expire. Currently we can't see any other way other than:

• Create a new database connection
• Create new models using the connection
• Replace all references of existing models across the code base to use the new models
• Retire the old connection

This gets complicated fast. The initial model references are retrieved after the initial database connection is established in different locations, so replacing them all with newer instances isn't as straight forward as it sounds. Furthermore, if there are db operations that are in progress at the time of establishing a new connection (for example a transaction), it would need to be aware of this, and keep using the old connection until ready to start using the new connection.

Is there a way of refreshing the connection with new credentials, without tearing everything down and re-creating the connection and models? (Refreshing the existing AWS credential expiry time isn't an option, as there is a maximum time these credentials can be refreshed).

Thanks, Michael

[Brendmanitix]

This appears to be something that's going to be handled by the MongoDb driver...eventually. https://jira.mongodb.org/browse/DRIVERS-1746

[gabrielmoreira]

@vkarpov15 I had the same problem. Until there is a proper way to provide fresh credentials to mongodb connections, our approach was to create a cache layer in our app side that expires before the STS credentials... so we generate new STS credentials, then we explicitly disconnect and connect mongoose again.

Something like this:

export async function connect(
  getMongoUri: () => Promise<string>,
): Promise<void> {
  // check if we have an active mongo connection from cache, or create a new one
  const hasActiveConnection = mongoActiveConnectionCache.get();
  if (!hasActiveConnection) {
    // force disconnect from mongodb before creating a new connection
    await disconnect();
    // create a new connection
    await createNewConnection(await getMongoUri());
    // save it to cache that we have an active mongodb connection
    mongoActiveConnectionCache.set(true);
  }
}

// getMongoUri will create new credentials and build a mongo uri with it. // createNewConnection uses mongoose.connect // disconnect is basically mongoose.disconnect

From my perspective, it would be perfect if mongoose and mongo driver accepts an async function to build credentials. So, whenever the driver needs, it would just invoke it. This function should not only return the credential, but also the expiration time, so it would the driver could watch it and disconnect any active connection after it expires, and reconnect by getting fresh credentials.

[florianbepunkt]

@gabrielmoreira Could you elaborate on your solution? How do you detect whether the sts session has expired or not? We are facing the same or a similar problem.

[gabrielmoreira]

@florianbepunkt , First, it's important to you to know that we run our code on AWS Lambda, so it only process one request at a time in the same lambda instance.

We don't actively check for connection expirations. But we do know that our assume role credentials expires about every hour. So, we renew it five minutes before it expires and then disconnect and reconnect to be safe. Just to avoid any connection issues.

The mongoActiveConnectionCache is like a clock that resets every 55 minutes. It doesn't remember the connection itself, just when it was created.

The disconnect also forcefully clears the cache. If you want, you could also observe relevant connection errors in your case and clear the cache, so the next connection request would force a create-credentials/disconnect/connect approach again.

Our code runs on AWS Lambda with a feature that keeps it ready to go, the provisioned concurrency. It's important to know that at the beginning of our Lambda event handler, we do call the connect function defined above, but as it has a cache mechanism, it won't recreate the connection on every request.

We also don't use the lambda available credentials (process.env.AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). If we did, we'd have to make specific access configurations for each Lambda on the MongoDB side, and as our infrastructures are managed separately (MongoDB vs. AWS Stack), it would be more error-prone to keep both in sync, especially for non-production ephemeral environments. Instead, we use a single role named something like 'mongo-connection-role', the only one we did configure on the MongoDB side, and on AWS, we only configure the role to only trust a few Lambdas to assume it.