Aurora Serverless v2

[autarchprinceps]

Name of the resource

AWS::RDS::DBInstance

Resource name

No response

Description

Aurora Serverless v2 is now GA. From how I understand the creation via UI/aws-cli the instances need to be of instance type db.serverless, which may or may not need to be added in CloudFormations property validation, but I definitively haven't found any way to specify the minimum and maximum ACUs yet. This is no longer a cluster setting, as with serverless v1, and even if, it now supports other values (e.g. 0.5) which definitively is neither documented as such or supported.

Other Details

https://aws.amazon.com/blogs/aws/amazon-aurora-serverless-v2-is-generally-available-instant-scaling-for-demanding-workloads/

[taspeotis]

I have used Cloudformation with db.serverless just fine - but there is no way to set min/max.

There is some suggestion from aws rds ... that ServerlessV2ScalingConfiguration exists for v2, replacing ScalingConfiguration.

[autarchprinceps]

@taspeotis I do not see, how you could have used CloudFormation just fine. It errors for me: Instance: Set the Serverless v2 scaling configuration on the parent DB cluster before creating a Serverless v2 DB instance. Cluster: Encountered unsupported property ServerlessV2ScalingConfiguration

I both need and cannot set the min & max values.

[taspeotis]

Again:

I have used Cloudformation with db.serverless just fine - but there is no way to set min/max.

You can configure the min and max in the web interface, and then use Cloudformation subsequently with the db.serverless engine version.

I did not say I could set ServerlessV2ScalingConfiguration. Again:

there is no way to set min/max.

[autarchprinceps]

"You can configure the min and max in the web interface" - only during initial cluster setup, which also creates the instances. Pressing modify on an existing DB Cluster without serverless instances, doesn't expose this setting, only modify to change the already set values, not setting it initially. Also combination of manual & CloudFormation steps isn't the goal of IaC. Clearly, ServerlessV2ScalingConfiguration needs to be added to CloudFormation DBCluster then, for Serverless v2 to be actually supported by CloudFormation. I expect, that then it should work, since db.serverless works for you on creating Instances. Apparently, there is an internal issue to add that, but no ETA as usual.

[taspeotis]

You can set them after the fact, modify one of the database instances and change it to serverless v2. And then you can run your cloudformation stack. I converted one instance to serverless v2 and that has the effect of setting min/max on the cluster. Then I submitted my cloudformation template and it set the rest of the instances to db.serverless.

I've migrated four clusters in place to serverless v2 with a combination of the web interface and cloudformation.

[cmesas]

I also have the problem to be able to define the configuration of the cluster. Any update with the resource structure to be able to specify the Max/Min ACUs in the cluster?

[kftsehk]

The closest thing now seem to be

1. Deploy AWS::RDS::DBCluster with EngineMode: provision
2. Run aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>, if you have not find this config, update the awscli
3. Deploy AWS::RDS::DBInstance with DBInstanceClass: db.serverless

[autarchprinceps]

https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1153

[dovidkopel]

+1

[arogozhnikov]

There are ~300 requests in CDK for serverless aurora (https://github.com/aws/aws-cdk/issues/20197), issue was given p1, but it is blocked by lack of support in cloudformation

[driverpt]

any ETA on this one?

[bilalq]

This issue is more upvoted than the vote count would suggest. Like someone mentioned earlier in the comments, #1153 is also effectively an issue covering the same need (just referencing a different specific CloudFormation type) and there's the CDK issue at https://github.com/aws/aws-cdk/issues/20197.

There are about 200 thumbs up here, about 100 in the other issue in the repo, and over 300 in the CDK one. I know there's likely to be some overlap in the thumbsup counts, but it's still a substantial ask from end customers.

I'm sure the CloudFormation team has heard this before, but it's really frustrating when a feature/service launches "GA" without CFN or CDK support.

[echennh-zz]

any ETA on this one?

@driverpt

Just heard back from AWS Support, and they said

Having said that, as you know any new feature or a fix requires the product to undergo rigorous testing and troubleshooting before it is made available, to ensure security and stability for end users. Unfortunately, I will not be able to provide you an ETA for it at the moment as the internal team does not share this information with us.

However, you also can track the Aurora Serverless v2 coverage via the CloudFormation GitHub Roadmap. There is currently an opened issue discussing this use case here [2][3].

[echennh-zz]

You can set them after the fact, modify one of the database instances and change it to serverless v2. And then you can run your cloudformation stack. I converted one instance to serverless v2 and that has the effect of setting min/max on the cluster. Then I submitted my cloudformation template and it set the rest of the instances to db.serverless.

I've migrated four clusters in place to serverless v2 with a combination of the web interface and cloudformation.

@taspeotis

Hi, can you clarify in more detail how you did this? Did you create the Aurora cluster in the console as a v2 cluster, then deploy a Cloudformation stack where one of the resources is the pre-existing cluster? If so, how did you do that? Up to this point, I've tried to update a stack via a CloudFormation template, but obviously since CF doesn't support Aurora v2, it just creates a new database cluster of type Aurora v1. Did you deploy a CF template and specify the SourceDBClusterIdentifier property or the SnapshotIdentifier to get it to look at the Aurora v2 cluster created in the console?

Any help appreciated!

[deuscapturus]

This hasn't even made it to the "Researching" phase of the roadmap. Why not? This issue seems to have fallen through the cracks. Could someone from AWS please provide a response? Thanks!

[gazal-k]

Interestingly, AWS Controllers for Kubernetes (ACK) already has support. https://aws-controllers-k8s.github.io/community/reference/rds/v1alpha1/dbcluster/ serverlessV2ScalingConfiguration. It uses the RDS APIs and does not depend on cloudformation support.

And terraform seems to support it as well. https://registry.terraform.io/modules/terraform-aws-modules/rds-aurora/aws/latest/examples/serverless. Not the first time I've seen terraform having better support for AWS resources than cloudformation.

[demiurg]

Guys, this would be great, currently have to make this from console instead of IaC.

[ccggeo]

Also bumping. This should definitely be native to CFN

[vincent-dm]

We are seriously blocked by this. Please prioritize it, thanks!

[shaheerkhan12605]

The closest thing now seem to be

1. Deploy AWS::RDS::DBCluster with EngineMode: provision
2. Run aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>, if you have not find this config, update the awscli
3. Deploy AWS::RDS::DBInstance with DBInstanceClass: db.serverless

@kftsehk I am using the following shell script to create the serverless v2

CLUSTER_NAME=$(cat rds_serverless.dev.json | jq -r .ClusterName)
NumberOfInstances=$(cat rds_serverless.dev.json | jq -r .NumberOfInstances) # 1 or 2
ENVIRONMENT=$(cat rds_serverless.dev.json | jq -r .Environment)
MinCapacity=$(cat rds_serverless.dev.json | jq -r .MinCapacity)
MaxCapacity=$(cat rds_serverless.dev.json | jq -r .MaxCapacity)

expr $NumberOfInstances + 0 # Converting to numeric

function tryexec() {
    "$@"
    retval=$?
    [[ $retval -eq 0 ]] && return 0
    echo "A command has failed:"
    echo "  $*"
    echo "Value returned: ${retval}"
   exit 254
}
tryexec aws rds --region $REGION modify-db-cluster --db-cluster-identifier $CLUSTER_NAME-cluster --serverless-v2-scaling-configuration MinCapacity=$MinCapacity,MaxCapacity=$MaxCapacity --output text
tryexec aws rds --region $REGION create-db-instance --db-instance-identifier $CLUSTER_NAME-primary-instance --db-cluster-identifier $CLUSTER_NAME-cluster --db-instance-class db.serverless --engine $engine --output text
tryexec aws rds --region $REGION wait db-instance-available --db-instance-identifier $CLUSTER_NAME-primary-instance --output text
if [[ $NumberOfInstances -gt 1 ]]; then
    tryexec aws rds --region $REGION create-db-instance --db-instance-identifier $CLUSTER_NAME-primary-instance-2 --db-cluster-identifier $CLUSTER_NAME-cluster --db-instance-class db.serverless --engine $engine --output text
    tryexec aws rds --region $REGION wait db-instance-available --db-instance-identifier $CLUSTER_NAME-primary-instance-2
fi
tryexec aws rds --region $REGION failover-db-cluster --db-cluster-identifier $CLUSTER_NAME-cluster --target-db-instance-identifier $CLUSTER_NAME-primary-instance --output text
tryexec aws rds --region $REGION delete-db-instance --db-instance-identifier $CLUSTER_NAME-primary --skip-final-snapshot --output text
tryexec aws rds --region $REGION wait db-instance-deleted --db-instance-identifier $CLUSTER_NAME-primary --output text

[matart15]

It is 2nd most :+1: issue. If we combine #1153 the it is the most :+1: issue.

not to mention CDK repo have same request with the most :+1: .

But it is not on public milestone

[mgirard772]

+1

[RajasGujarathi]

The closest thing now seem to be

1. Deploy `AWS::RDS::DBCluster` with `EngineMode: provision`

2. Run `aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>`, if you have not find this config, update the `awscli`

3. Deploy `AWS::RDS::DBInstance` with `DBInstanceClass: db.serverless`

Thanks, this worked smoothly 🙏

[Beast12]

For Anyone that is trying to put this all in cloudformation...

CommandRunnerServerlessV2:
    DependsOn: RDSClusterServerlessV2
    Condition: IsServerless-v2
    Type: AWSUtility::CloudFormation::CommandRunner
    Properties:
      Command: !Sub |
        sudo rm -rf /usr/local/aws && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && sudo ./aws/install && \
        aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":8}' --db-cluster-identifier ${RDSClusterServerlessV2} --region ${AWS::Region} \
        > /command-output.txt
      LogGroup: CloudRunner
      SubnetId:
        !Select [0, !FindInMap [Environments, !Ref Environment, PrivateSubnets]]
      Role: !Ref CommandRunnerRole
      InstanceType: t2.micro

Make sure the Role (Which refers to the instance profile btw...) has enough permissions to run the command of course

[ekeyser]

any ETA on this one?

@driverpt

Just heard back from AWS Support, and they said

Having said that, as you know any new feature or a fix requires the product to undergo rigorous testing and troubleshooting before it is made available, to ensure security and stability for end users. Unfortunately, I will not be able to provide you an ETA for it at the moment as the internal team does not share this information with us.

However, you also can track the Aurora Serverless v2 coverage via the CloudFormation GitHub Roadmap. There is currently an opened issue discussing this use case here [2][3].

The same standard reply from support re no ETA. How surprising.

BTW, this response about rigorous testing is complete garbage but I'm sure plenty of people will just lap it up as a valid excuse. They need to take responsibility and admit that releasing a service or product involves coordination with other teams and testing beforehand. This is what it means to be a professional company and a professional software development organization and their rhetoric surrounding this is dogsh*t. What they're doing instead is releasing products in the market in hopes of obtaining a market advantage but which are not integrated with the rest of the AWS service offerings and tooling. This in turn just makes everyone's lives supporting these half-baked offerings miserable.

This is the hallmark of a company that is no longer a gold standard and which is simply just churning out garbage.

[relsunkaev]

@ekeyser

There is actually a soft ETA, as per this reply.

[skysb]

Seems like this has been implemented already right? https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html

At the section Creating an Amazon Aurora Serverless v2 DB cluster ?? or am I mistaken?

[zsims]

You're right @skysb, seems it was implemented and this issue hasn't been updated.

I tried it out and can confirm it works well 👍

[moltar]

woohoo party!

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-serverlessv2scalingconfiguration

[osdrv]

Hey all, the feature is now available globally. Please refer to the doc for more details. Please open a new ticket upon any other issue concerning ServerlessV2. This ticket might be closed now.