Aurora v1 Environment Addon

[dannyrandall]

Discussed in https://github.com/aws/copilot-cli/discussions/4943

Running copilot storage init --lifecycle environment --storage-type Aurora --serverless-version v1 generates an addon using Aurora v2. Copilot should generate an addon using Aurora v1 in this case!

[TylerBoni]

Also having this issue. Is V1 no longer being supported by copilot?

[KollaAdithya]

Hello @TylerBoni !

Copilot generates a Aurora v1 template at workload level ✅ copilot storage init --lifecycle workload --storage-type Aurora --serverless-version v1

Even after specifying v1 for flag --serverless-version, at the env level Copilot does not generate a Aurora V1 template, instead it generates a Aurora v2 template.

[TylerBoni]

Thanks @KollaAdithya. I see this now. It would be great to have this feature, or at least the steps to manually adjust the addon manifest.. At an absolute minimum, it would be nice to let the user know that their flag will be ignored. Thanks for the reply.

[morganney]

@dannyrandall any update on this at all? I was about to switch from Aurora v2 to v1 (to avoid the expensive $40/mo. cost of v2 at rest) but luckily I found this issue. I am on copilot v1.32.1 and do not want to have to downgrade to v1.22 as suggested in this discussion just in order to be able to use Aurora v1 with copilot.

How did the copilot docs get updated without anybody testing actual support for --serverless-version v1? Please remove this from the docs and CLI help until users can actually use v1:

      --serverless-version string   Optional. Aurora Serverless version.
                                    Must be either "v1" or "v2" (default "v2").

If I have to control aspects of AWS infrastructure outside of copilot, then copilot loses some of its appeal.

[Lou1415926]

@morganney sorry for the confusion and the trouble that's caused!

I will go ahead and make --serverless-version v1 as a viable option for environment level storage. https://github.com/aws/copilot-cli/pull/5612/commits is not a complete implementation, but it contains the template that Copilot would give you after you've run copilot storage init --serverless-version v1 --lifecycle environment.

Before this is released, you can:

1. Copy and paste the content of addons.parameters.yml to the file environments/addons/addons.parameters.yml.
2. Copy and paste the following content to environments/addons/serverlessv1.yml (the filename "serverlessv1.yml" is arbitrary - it can be whatever you like) :

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  # Customize your Aurora Serverless cluster by setting the default value of the following parameters.
  <placeholder:clustername>DBName:
    Type: String
    Description: The name of the initial database to be created in the DB cluster.
    Default: <placeholder:dbname>
    # Cannot have special characters
    # Naming constraints: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.Constraints
  VPCID:
    Type: String
    Description: The ID of the VPC in which to create the Aurora Serverless cluster.
    Default: ""
  PrivateSubnets:
    Type: String
    Description: The IDs of the private subnets in which to create the Aurora Serverless cluster.
    Default: ""
Mappings:
  <placeholder:clustername>EnvScalingConfigurationMap:
    test:
      "DBMinCapacity": 1 # AllowedValues: [1, 2, 4, 8, 16, 32, 64, 128, 256]
      "DBMaxCapacity": 8 # AllowedValues: [1, 2, 4, 8, 16, 32, 64, 128, 256]
    All:
      "DBMinCapacity": 1 # AllowedValues: [1, 2, 4, 8, 16, 32, 64, 128, 256]
      "DBMaxCapacity": 8 # AllowedValues: [1, 2, 4, 8, 16, 32, 64, 128, 256]

Resources:
  <placeholder:clustername>DBSubnetGroup:
    Type: 'AWS::RDS::DBSubnetGroup'
    Properties:
      DBSubnetGroupDescription: Group of Copilot private subnets for Aurora cluster.
      SubnetIds:
        !Split [',', !Ref PrivateSubnets]
  <placeholder:clustername>WorkloadSecurityGroup:
    Metadata:
      'aws:copilot:description': 'A security group for one or more workloads to access the DB cluster <placeholder:clustername>'
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Sub 'The security group to access DB cluster <placeholder:clustername>.'
      VpcId: !Ref VPCID
      Tags:
        - Key: Name
          Value: !Sub 'copilot-${App}-${Env}-Aurora'
  <placeholder:clustername>DBClusterSecurityGroup:
    Metadata:
      'aws:copilot:description': 'A security group for your DB cluster <placeholder:clustername>'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: The Security Group for the database cluster.
      SecurityGroupIngress:
        - ToPort: 3306
          FromPort: 3306
          IpProtocol: tcp
          Description: Ingress from one or more workloads in the environment.
          SourceSecurityGroupId: !Ref <placeholder:clustername>WorkloadSecurityGroup
      VpcId: !Ref VPCID
      Tags:
        - Key: Name
          Value: !Sub 'copilot-${App}-${Env}-Aurora'
  <placeholder:clustername>AuroraSecret:
    Metadata:
      'aws:copilot:description': 'A Secrets Manager secret to store your DB credentials'
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: !Sub Aurora main user secret for ${AWS::StackName}
      GenerateSecretString:
        SecretStringTemplate: '{"username": "admin"}'
        GenerateStringKey: "password"
        ExcludePunctuation: true
        IncludeSpace: false
        PasswordLength: 16

  <placeholder:clustername>DBClusterParameterGroup:
    Metadata:
      'aws:copilot:description': 'A DB parameter group for engine configuration values'
    Type: 'AWS::RDS::DBClusterParameterGroup'
    Properties:
      Description: !Ref 'AWS::StackName'
      Family: 'aurora-mysql5.7'
      Parameters:
        character_set_client: 'utf8'
  <placeholder:clustername>DBCluster:
    Metadata:
      'aws:copilot:description': 'The <placeholder:clustername> Aurora Serverless database cluster'
    Type: 'AWS::RDS::DBCluster'
    Properties:
      MasterUsername:
        !Join [ "",  [ '{{resolve:secretsmanager:', !Ref <placeholder:clustername>AuroraSecret, ":SecretString:username}}" ]]
      MasterUserPassword:
        !Join [ "",  [ '{{resolve:secretsmanager:', !Ref <placeholder:clustername>AuroraSecret, ":SecretString:password}}" ]]
      DatabaseName: !Ref <placeholder:clustername>DBName
      Engine: 'aurora-mysql'
      EngineVersion: '5.7.mysql_aurora.2.07.1'
      EngineMode: serverless
      DBClusterParameterGroupName: !Ref <placeholder:clustername>DBClusterParameterGroup
      DBSubnetGroupName: !Ref <placeholder:clustername>DBSubnetGroup
      VpcSecurityGroupIds:
        - !Ref <placeholder:clustername>DBClusterSecurityGroup
      ScalingConfiguration:
        AutoPause: true
        # Replace "All" below with "!Ref Env" to set different autoscaling limits per environment.
        MinCapacity: !FindInMap [<placeholder:clustername>EnvScalingConfigurationMap, All, DBMinCapacity]
        MaxCapacity: !FindInMap [<placeholder:clustername>EnvScalingConfigurationMap, All, DBMaxCapacity]
        SecondsUntilAutoPause: 1000
  <placeholder:clustername>SecretAuroraClusterAttachment:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId: !Ref <placeholder:clustername>AuroraSecret
      TargetId: !Ref <placeholder:clustername>DBCluster
      TargetType: AWS::RDS::DBCluster
Outputs:
  <placeholder:clustername>Secret: # injected as {{envVarSecret .ClusterName | toSnakeCase}} environment variable by Copilot.
    Description: "The JSON secret that holds the database username and password. Fields are 'host', 'port', 'dbname', 'username', 'password', 'dbClusterIdentifier' and 'engine'"
    Value: !Ref <placeholder:clustername>AuroraSecret
    Export:
      Name: !Sub ${App}-${Env}-<placeholder:clustername>AuroraSecret
  <placeholder:clustername>SecurityGroup:
    Description: "The security group to attach to the workload."
    Value: !Ref <placeholder:clustername>WorkloadSecurityGroup
    Export:
      Name: !Sub ${App}-${Env}-<placeholder:clustername>SecurityGroup

This should let you create an aurora serverless v1 cluster as an environment addon.

[morganney]

@Lou1415926 thank you for the very helpful, and detailed response. I appreciate your willingness to help and add additional support for v1 environment lifecycle 🙏 .

https://github.com/aws/copilot-cli/pull/5612/commits is not a complete implementation

I look forward to --lifecycle environment being supported under an official copilot release. Until then, I have decided to use --serverless-version v1 --lifecycle workload for my test environment. This approach is supported by my version of copilot v1.32.1.

Would you happen to know the DBMinCapacity allowed for a v1 workload lifecycle Aurora cluster? I'm trying to scale down as much as possible and have noticed some discrepancies in the config values generated by the templates based on the options provided. Figured I'd ask before I try some values and see what works.

[Lou1415926]

Would you happen to know the DBMinCapacity allowed for a v1 workload lifecycle Aurora cluster?

It should be 1 for MySQL, and 2 for PostgreSQL! (see here for more)

[morganney]

Thanks, I saw the documentation but wanted to know if the docs were possibly outdated and I could scale down to 0.5 or 1 (I tried, and you can't).

@Lou1415926 do you know if v2 plans to support scaling down to zero, like v1?

I recently learnt that v1 will be end of life December 2024:

We are reaching out to let you know that as of December 31, 2024, Amazon Aurora will no longer support Serverless version 1 (v1). As per the Aurora Version Policy [1], we are providing 12 months notice to give you time to upgrade your database cluster(s). Aurora supports two versions of Serverless. We are only announcing the end of support for Serverless v1. Aurora Serverless v2 continues to be supported. We recommend that you proactively upgrade your databases running Amazon Aurora Serverless v1 to Amazon Aurora Serverless v2 at your convenience before December 31, 2024.

This isn't good, and there will be lots of unhappy customers without the ability to scale v2 down to 0, particularly for testing and staging environments.

Sorry for somewhat spamming the thread, thought it would be relevant since this is about a v1 addon...

[Lou1415926]

do you know if v2 plans to support scaling down to zero, like v1?

I don't have any new information for now 😞 I can forward the concern for you!

On a side note though, from what I've heard after scaling to zero (by basically shutting down the instance), v1 does take up to 10-20 seconds to spin back up. I am not sure if this was acceptable for your workloads - if not, then v1 probably wouldn't have been perfect for you in the first place 💭

[morganney]

v1 does take up to 10-20 seconds to spin back up. I am not sure if this was acceptable for your workloads

This is acceptable, at least much more so than the v2 base pricing 😀. Thanks for passing on the concern.