provider-aws
provider-aws copied to clipboard
Published
20 hours ago •
crossplane-contrib
crossplane-contrib
Huge number of API Calls
What happened?
Hello, we found out that most of the calls logged by Cloudtrail are from provider-aws, and it significantly impacts our Guarduty bill.
After some digging we tried the solution from #847 and added --poll=5m, which drastically reduced the calls number overall, however we think that the call rate is still very high, considering our setup.
Here is the summary of the last hour calls, about 80% of AssumeRole, and 100%-ish of ModifyDBInstance and AddTagsToResource are from crossplane-provider-aws.
'AddTagsToResource': 1812,
'AssignPrivateIpAddresses': 59,
'AssumeRole': 8662,
'AssumeRoleWithWebIdentity': 55,
'ModifyDBInstance': 2015,
'ModifyDBParameterGroup': 66,
'PutBucketAcl': 26,
'PutBucketPolicy': 26,
'PutEvaluations': 77,
'UnassignPrivateIpAddresses': 59,
'UpdateInstanceInformation': 275
We have a EKS cluster and are starting to migrate our clients from metal and we currently have created a small amount of RDS instances (we expect to have at least 3x this number), so we are affraid of an exponential increase of the calls to AWS in the next weeks :
$ kubectl get rdsinstances.database.aws.crossplane.io -A | wc -l
26 # all are Ready, only ONE is not Synced
Playing around with max-reconcile-rate didn't change anything and sync default to 1 hour shouldn't have much impact.
- Is there any other ways to reduce the calls number ?
- We could increase again the
pollargument but we are not sure if this is a good idea to reduce the crossplane reactivity - Why the provider keeps rewriting the tags using
AddTagsToResource? (there is no changes to the tags once the instance is created) - 2000
ModifyDBInstanceper hour for 26 RDS Instances seems a bit intense IMO
Thanks in advance and let me know if you need more informations.
How can we reproduce it?
Standard install of crossplane + provider-aws, create a bunch of rds instances and check cloudtrail
What environment did it happen in?
- Crossplane version:
1.7.0 - provider-aws :
0.26.0 - Kubernetes version (use
kubectl version)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.13-eks-84b4fe6", GitCommit:"e1318dce57b3e319a2e3fecf343677d1c4d4aa75", GitTreeState:"clean", BuildDate:"2022-06-09T18:22:07Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes distribution :
EKS 1.21.5
Might be related to #828 - but I don't understand how you have only so many AssumeRole and so few AssumeRoleWithWebIdentity. What kind of ProviderConfig do you use?
Could you also take a look into what kind of resource is tagged with AddTagsToResource? Can you then do kubectl get -w -o yaml on the managed resource and see what changes?
Re. RDSInstance, I wonder if you could upgrade to 0.29 to check if the situation has improved. The only issue I know of now is if you use UpperCase in maintenance/backup window, AWS uses lower case.
Hello, thanks for your answer, let me explain more of our setup :
What kind of ProviderConfig do you use?
We have a multi-tenant cluster and to avoid any configuration mistakes from our clients, we made our own Kinds available as Composition using CompositeResourceDefinition, there is one for MySQL (Kind : MysqlInstance) and one for PostgreSQL (Kind : PostgresInstance).
You can find below the ones for MySQL (postgres is basically the same, only the values changes), note that I replaced our company name by "company".
MysqlInstance manifest example
## Note : the tags are added by a Kyverno mutation
## only when the operation is CREATE for the object
## same for the subnets ids and vpc id
apiVersion: database.company.org/v1alpha1
kind: MysqlInstance
metadata:
name: sample-app-staging-rdsmysql-mysqlinstance
namespace: sample-app-staging
spec:
compositionRef:
name: xmysqlinstances.aws.database.company.org
compositionUpdatePolicy: Automatic
parameters:
dbParameterGroup:
forProvider:
dbParameterGroupFamily: mysql8.0
description: dbParameterGroup
parameters:
- applyMethod: immediate
parameterName: transaction_isolation
parameterValue: READ-COMMITTED
region: us-east-1
tags:
- key: kubernetes_cluster
value: eks01-cluster
- key: kubernetes_namespace
value: sample-app-staging
dbSubnetGroup:
forProvider:
description: dbSubnetGroup
region: us-east-1
subnetIds:
- subnet-xxxxxxxxxxxxxxxxx # modified
- subnet-yyyyyyyyyyyyyyyyy # modified
- subnet-zzzzzzzzzzzzzzzzz # modified
tags:
- key: kubernetes_cluster
value: eks01-cluster
- key: kubernetes_namespace
value: sample-app-staging
rdsInstance:
forProvider:
allocatedStorage: 10
applyModificationsImmediately: true
autoMinorVersionUpgrade: true
backupRetentionPeriod: 0
copyTagsToSnapshot: true
dbInstanceClass: db.t4g.small
dbName: ""
dbSubnetGroupNameSelector:
matchControllerRef: true
deletionProtection: false
engine: mysql
engineVersion: 8.0.28
masterUsername: root
maxAllocatedStorage: 20
multiAZ: false
region: us-east-1
skipFinalSnapshotBeforeDeletion: true
storageEncrypted: true
storageType: gp2
tags:
- key: kubernetes_cluster
value: eks01-cluster
- key: kubernetes_namespace
value: sample-app-staging
vpcSecurityGroupIDSelector:
matchControllerRef: true
securityGroup:
forProvider:
description: securityGroup
egress:
- fromPort: 3306
ipProtocol: tcp
ipRanges:
- cidrIp: 10.0.0.0/8
toPort: 3306
groupName: ""
ingress:
- fromPort: 3306
ipProtocol: tcp
ipRanges:
- cidrIp: 10.0.0.0/8
toPort: 3306
region: us-east-1
tags:
- key: kubernetes_cluster
value: eks01-cluster
- key: kubernetes_namespace
value: sample-app-staging
vpcId: vpc-0000000000000000 # modified
resourceRef:
apiVersion: database.company.org/v1alpha1
kind: XMysqlInstance
name: sample-app-staging-rdsmysql-mysqlinstance-5q9dz
writeConnectionSecretToRef:
name: sample-app-staging-rdsmysql-db
Composition
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: xmysqlinstances.aws.database.company.org
labels:
provider: aws
service: rds
spec:
writeConnectionSecretsToNamespace: crossplane-system
compositeTypeRef:
apiVersion: database.company.org/v1alpha1
kind: XMysqlInstance
resources:
- name: db-subnet-group
base:
apiVersion: database.aws.crossplane.io/v1beta1
kind: DBSubnetGroup
spec:
forProvider: {}
providerConfigRef:
name: providerconfig-aws
patches:
- fromFieldPath: "spec.parameters.dbSubnetGroup.forProvider"
toFieldPath: "spec.forProvider"
- type: ToCompositeFieldPath
fromFieldPath: status.conditions
toFieldPath: status.components.dbSubnetGroup
- name: security-group
base:
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
labels:
type: securityGroup
spec:
forProvider: {}
providerConfigRef:
name: providerconfig-aws
patches:
- fromFieldPath: "spec.parameters.securityGroup.forProvider"
toFieldPath: "spec.forProvider"
- fromFieldPath: "metadata.name"
toFieldPath: "spec.forProvider.groupName"
- type: ToCompositeFieldPath
fromFieldPath: status.conditions
toFieldPath: status.components.securityGroup
- name: dbParameterGroupName
base:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBParameterGroup
metadata:
labels:
type: dbParameterGroup
spec:
forProvider: {}
providerConfigRef:
name: providerconfig-aws
patches:
- fromFieldPath: "spec.parameters.dbParameterGroup.forProvider"
toFieldPath: "spec.forProvider"
- type: ToCompositeFieldPath
fromFieldPath: status.conditions
toFieldPath: status.components.dbParameterGroup
- name: rdsinstance
base:
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
spec:
forProvider:
writeConnectionSecretToRef:
namespace: crossplane-system
providerConfigRef:
name: providerconfig-aws
patches:
- fromFieldPath: "spec.parameters.rdsInstance.forProvider"
toFieldPath: "spec.forProvider"
- fromFieldPath: "metadata.uid"
toFieldPath: "spec.writeConnectionSecretToRef.name"
transforms:
- type: string
string:
fmt: "%s-mysql"
- fromFieldPath: "spec.resourceRefs[2].name"
toFieldPath: "spec.forProvider.dbParameterGroupName"
- type: ToCompositeFieldPath
fromFieldPath: status.conditions
toFieldPath: status.components.rdsInstance
- type: ToCompositeFieldPath
fromFieldPath: "metadata.name"
toFieldPath: "status.databaseInstanceName"
connectionDetails:
- fromConnectionSecretKey: username
- fromConnectionSecretKey: password
- fromConnectionSecretKey: endpoint
- fromConnectionSecretKey: port
- fromFieldPath: spec.forProvider.dbName
name: dbName
CompositeResourceDefinition
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xmysqlinstances.database.company.org
spec:
group: database.company.org
names:
kind: XMysqlInstance
plural: xmysqlinstances
claimNames:
kind: MysqlInstance
plural: mysqlinstances
shortNames:
- mysql
connectionSecretKeys:
- username
- password
- endpoint
- port
- dbName
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
parameters:
type: object
properties:
# -------------------------------------------------------------
dbSubnetGroup:
type: object
properties:
forProvider:
required:
- description
type: object
description: dbSubnetGroup
properties:
description:
description: The description for the DB subnet group.
type: string
region:
description: Region is the region you'd like your DBSubnetGroup
to be created in.
type: string
subnetIdRefs:
description: SubnetIDRefs is a set of references that each retrieve
the subnetID from the referenced Subnet
items:
description: A Reference to a named object.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
type: array
subnetIdSelector:
description: SubnetIDSelector selects a set of references that
each retrieve the subnetID from the referenced Subnet
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
subnetIds:
description: The EC2 Subnet IDs for the DB subnet group.
items:
type: string
type: array
tags:
description: A list of tags. For more information, see Tagging
Amazon RDS Resources (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html)
in the Amazon RDS User Guide.
items:
description: Tag is a metadata assigned to an Amazon RDS resource
consisting of a key-value pair. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/Tag
properties:
key:
description: 'A key is the required name of the tag. The
string value can be from 1 to 128 Unicode characters in
length and can''t be prefixed with "aws:" or "rds:". The
string can only contain only the set of Unicode letters,
digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'',
''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").'
type: string
value:
description: 'A value is the optional value of the tag.
The string value can be from 1 to 256 Unicode characters
in length and can''t be prefixed with "aws:" or "rds:".
The string can only contain only the set of Unicode letters,
digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'',
''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").'
type: string
type: object
type: array
# -------------------------------------------------------------
securityGroup:
type: object
properties:
forProvider:
required:
- description
- groupName
type: object
description: SecurityGroupParameters define the desired state of an
AWS VPC Security Group.
properties:
description:
description: A description of the security group.
type: string
egress:
description: '[EC2-VPC] One or more outbound rules associated
with the security group.'
items:
description: IPPermission Describes a set of permissions for
a security group rule.
properties:
fromPort:
description: The start of port range for the TCP and UDP
protocols, or an ICMP/ICMPv6 type number. A value of -1
indicates all ICMP/ICMPv6 types. If you specify all ICMP/ICMPv6
types, you must specify all codes.
format: int32
type: integer
ipProtocol:
description: "The IP protocol name (tcp, udp, icmp, icmpv6)
or number (see Protocol Numbers (http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml)).
\n [VPC only] Use -1 to specify all protocols. When authorizing
security group rules, specifying -1 or a protocol number
other than tcp, udp, icmp, or icmpv6 allows traffic on
all ports, regardless of any port range you specify. For
tcp, udp, and icmp, you must specify a port range. For
icmpv6, the port range is optional; if you omit the port
range, traffic for all types and codes is allowed."
type: string
ipRanges:
description: The IPv4 ranges.
items:
description: IPRange describes an IPv4 range.
properties:
cidrIp:
description: The IPv4 CIDR range. You can either specify
a CIDR range or a source security group, not both.
To specify a single IPv4 address, use the /32 prefix
length.
type: string
description:
description: "A description for the security group
rule that references this IPv4 address range. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*"
type: string
required:
- cidrIp
type: object
type: array
ipv6Ranges:
description: "The IPv6 ranges. \n [VPC only]"
items:
description: IPv6Range describes an IPv6 range.
properties:
cidrIPv6:
description: The IPv6 CIDR range. You can either specify
a CIDR range or a source security group, not both.
To specify a single IPv6 address, use the /128 prefix
length.
type: string
description:
description: "A description for the security group
rule that references this IPv6 address range. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*"
type: string
required:
- cidrIPv6
type: object
type: array
prefixListIds:
description: "PrefixListIDs for an AWS service. With outbound
rules, this is the AWS service to access through a VPC
endpoint from instances associated with the security group.
\n [VPC only]"
items:
description: PrefixListID describes a prefix list ID.
properties:
description:
description: "A description for the security group
rule that references this prefix list ID. \n Constraints:
Up to 255 characters in length. Allowed characters
are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*"
type: string
prefixListId:
description: The ID of the prefix.
type: string
required:
- prefixListId
type: object
type: array
toPort:
description: The end of port range for the TCP and UDP protocols,
or an ICMP/ICMPv6 code. A value of -1 indicates all ICMP/ICMPv6
codes. If you specify all ICMP/ICMPv6 types, you must
specify all codes.
format: int32
type: integer
userIdGroupPairs:
description: UserIDGroupPairs are the source security group
and AWS account ID pairs. It contains one or more accounts
and security groups to allow flows from security groups
of other accounts.
items:
description: UserIDGroupPair describes a security group
and AWS account ID pair.
properties:
description:
description: "A description for the security group
rule that references this user ID group pair. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*"
type: string
groupId:
description: The ID of the security group.
type: string
groupIdRef:
description: GroupIDRef reference a security group
to retrieve its GroupID
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
groupIdSelector:
description: GroupIDSelector selects reference to
a security group to retrieve its GroupID
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with
matching labels is selected.
type: object
type: object
groupName:
description: "The name of the security group. In a
request, use this parameter for a security group
in EC2-Classic or a default VPC only. For a security
group in a nondefault VPC, use the security group
ID. \n For a referenced security group in another
VPC, this value is not returned if the referenced
security group is deleted."
type: string
userId:
description: "The ID of an AWS account. \n For a referenced
security group in another VPC, the account ID of
the referenced security group is returned in the
response. If the referenced security group is deleted,
this value is not returned. \n [EC2-Classic] Required
when adding or removing rules that reference a security
group in another AWS account."
type: string
vpcId:
description: The ID of the VPC for the referenced
security group, if applicable.
type: string
vpcIdRef:
description: VPCIDRef reference a VPC to retrieve
its vpcId
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
vpcIdSelector:
description: VPCIDSelector selects reference to a
VPC to retrieve its vpcId
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with
matching labels is selected.
type: object
type: object
vpcPeeringConnectionId:
description: The ID of the VPC peering connection,
if applicable.
type: string
type: object
type: array
required:
- ipProtocol
type: object
type: array
groupName:
description: The name of the security group.
type: string
ingress:
description: One or more inbound rules associated with the security
group.
items:
description: IPPermission Describes a set of permissions for
a security group rule.
properties:
fromPort:
description: The start of port range for the TCP and UDP
protocols, or an ICMP/ICMPv6 type number. A value of -1
indicates all ICMP/ICMPv6 types. If you specify all ICMP/ICMPv6
types, you must specify all codes.
format: int32
type: integer
ipProtocol:
description: "The IP protocol name (tcp, udp, icmp, icmpv6)
or number (see Protocol Numbers (http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml)).
\n [VPC only] Use -1 to specify all protocols. When authorizing
security group rules, specifying -1 or a protocol number
other than tcp, udp, icmp, or icmpv6 allows traffic on
all ports, regardless of any port range you specify. For
tcp, udp, and icmp, you must specify a port range. For
icmpv6, the port range is optional; if you omit the port
range, traffic for all types and codes is allowed."
type: string
ipRanges:
description: The IPv4 ranges.
items:
description: IPRange describes an IPv4 range.
properties:
cidrIp:
description: The IPv4 CIDR range. You can either specify
a CIDR range or a source security group, not both.
To specify a single IPv4 address, use the /32 prefix
length.
type: string
description:
description: "A description for the security group
rule that references this IPv4 address range. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*"
type: string
required:
- cidrIp
type: object
type: array
ipv6Ranges:
description: "The IPv6 ranges. \n [VPC only]"
items:
description: IPv6Range describes an IPv6 range.
properties:
cidrIPv6:
description: The IPv6 CIDR range. You can either specify
a CIDR range or a source security group, not both.
To specify a single IPv6 address, use the /128 prefix
length.
type: string
description:
description: "A description for the security group
rule that references this IPv6 address range. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=&;{}!$*"
type: string
required:
- cidrIPv6
type: object
type: array
prefixListIds:
description: "PrefixListIDs for an AWS service. With outbound
rules, this is the AWS service to access through a VPC
endpoint from instances associated with the security group.
\n [VPC only]"
items:
description: PrefixListID describes a prefix list ID.
properties:
description:
description: "A description for the security group
rule that references this prefix list ID. \n Constraints:
Up to 255 characters in length. Allowed characters
are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*"
type: string
prefixListId:
description: The ID of the prefix.
type: string
required:
- prefixListId
type: object
type: array
toPort:
description: The end of port range for the TCP and UDP protocols,
or an ICMP/ICMPv6 code. A value of -1 indicates all ICMP/ICMPv6
codes. If you specify all ICMP/ICMPv6 types, you must
specify all codes.
format: int32
type: integer
userIdGroupPairs:
description: UserIDGroupPairs are the source security group
and AWS account ID pairs. It contains one or more accounts
and security groups to allow flows from security groups
of other accounts.
items:
description: UserIDGroupPair describes a security group
and AWS account ID pair.
properties:
description:
description: "A description for the security group
rule that references this user ID group pair. \n
Constraints: Up to 255 characters in length. Allowed
characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*"
type: string
groupId:
description: The ID of the security group.
type: string
groupIdRef:
description: GroupIDRef reference a security group
to retrieve its GroupID
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
groupIdSelector:
description: GroupIDSelector selects reference to
a security group to retrieve its GroupID
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with
matching labels is selected.
type: object
type: object
groupName:
description: "The name of the security group. In a
request, use this parameter for a security group
in EC2-Classic or a default VPC only. For a security
group in a nondefault VPC, use the security group
ID. \n For a referenced security group in another
VPC, this value is not returned if the referenced
security group is deleted."
type: string
userId:
description: "The ID of an AWS account. \n For a referenced
security group in another VPC, the account ID of
the referenced security group is returned in the
response. If the referenced security group is deleted,
this value is not returned. \n [EC2-Classic] Required
when adding or removing rules that reference a security
group in another AWS account."
type: string
vpcId:
description: The ID of the VPC for the referenced
security group, if applicable.
type: string
vpcIdRef:
description: VPCIDRef reference a VPC to retrieve
its vpcId
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
vpcIdSelector:
description: VPCIDSelector selects reference to a
VPC to retrieve its vpcId
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with
matching labels is selected.
type: object
type: object
vpcPeeringConnectionId:
description: The ID of the VPC peering connection,
if applicable.
type: string
type: object
type: array
required:
- ipProtocol
type: object
type: array
region:
description: Region is the region you'd like your SecurityGroup
to be created in.
type: string
tags:
description: Tags represents to current ec2 tags.
items:
description: Tag defines a tag
properties:
key:
description: Key is the name of the tag.
type: string
value:
description: Value is the value of the tag.
type: string
required:
- key
- value
type: object
type: array
vpcId:
description: VPCID is the ID of the VPC.
type: string
vpcIdRef:
description: VPCIDRef references a VPC to and retrieves its vpcId
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
vpcIdSelector:
description: VPCIDSelector selects a reference to a VPC to and
retrieves its vpcId
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
# -------------------------------------------------------------
dbParameterGroup:
type: object
properties:
forProvider:
required:
- dbParameterGroupFamily
- description
- region
type: object
description: DBParameterGroupParameters defines the desired state
of DBParameterGroup
properties:
dbParameterGroupFamily:
description: "The DB parameter group family name. A DB parameter
group can be associated with one and only one DB parameter group
family, and can be applied only to a DB instance running a database
engine and engine version compatible with that DB parameter
group family. \n To list all of the available parameter group
families for a DB engine, use the following command: \n aws
rds describe-db-engine-versions --query \"DBEngineVersions[].DBParameterGroupFamily\"
--engine <engine> \n For example, to list all of the available
parameter group families for the MySQL DB engine, use the following
command: \n aws rds describe-db-engine-versions --query \"DBEngineVersions[].DBParameterGroupFamily\"
--engine mysql \n The output contains duplicates. \n The following
are the valid DB engine values: \n * aurora (for MySQL 5.6-compatible
Aurora) \n * aurora-mysql (for MySQL 5.7-compatible Aurora)
\n * aurora-postgresql \n * mariadb \n * mysql \n * oracle-ee
\n * oracle-ee-cdb \n * oracle-se2 \n * oracle-se2-cdb \n *
postgres \n * sqlserver-ee \n * sqlserver-se \n * sqlserver-ex
\n * sqlserver-web"
type: string
description:
description: The description for the DB parameter group.
type: string
parameters:
description: A list of parameters to associate with this DB parameter
group
items:
properties:
allowedValues:
type: string
applyMethod:
type: string
applyType:
type: string
dataType:
type: string
description:
type: string
isModifiable:
type: boolean
minimumEngineVersion:
type: string
parameterName:
type: string
parameterValue:
type: string
source:
type: string
supportedEngineModes:
items:
type: string
type: array
type: object
type: array
region:
description: Region is which region the DBParameterGroup will
be created.
type: string
tags:
description: Tags to assign to the DB parameter group.
items:
properties:
key:
type: string
value:
type: string
type: object
type: array
# -------------------------------------------------------------
rdsInstance:
type: object
properties:
forProvider:
required:
- dbInstanceClass
- engine
type: object
description: RDSInstanceParameters define the desired state of an
AWS Relational Database Service instance.
properties:
allocatedStorage:
description: 'AllocatedStorage is the amount of storage (in gibibytes)
to allocate for the DB instance. Type: Integer Amazon Aurora
Not applicable. Aurora cluster volumes automatically grow as
the amount of data in your database increases, though you are
only charged for the space that you use in an Aurora cluster
volume. MySQL Constraints to the amount of storage for each
storage type are the following: * General Purpose (SSD) storage
(gp2): Must be an integer from 20 to 16384. * Provisioned IOPS
storage (io1): Must be an integer from 100 to 16384. * Magnetic
storage (standard): Must be an integer from 5 to 3072. MariaDB
Constraints to the amount of storage for each storage type are
the following: * General Purpose (SSD) storage (gp2): Must be
an integer from 20 to 16384. * Provisioned IOPS storage (io1):
Must be an integer from 100 to 16384. * Magnetic storage (standard):
Must be an integer from 5 to 3072. PostgreSQL Constraints to
the amount of storage for each storage type are the following:
* General Purpose (SSD) storage (gp2): Must be an integer from
20 to 16384. * Provisioned IOPS storage (io1): Must be an integer
from 100 to 16384. * Magnetic storage (standard): Must be an
integer from 5 to 3072. Oracle Constraints to the amount of
storage for each storage type are the following: * General Purpose
(SSD) storage (gp2): Must be an integer from 20 to 16384. *
Provisioned IOPS storage (io1): Must be an integer from 100
to 16384. * Magnetic storage (standard): Must be an integer
from 10 to 3072. SQL Server Constraints to the amount of storage
for each storage type are the following: * General Purpose (SSD)
storage (gp2): Enterprise and Standard editions: Must be an
integer from 200 to 16384. Web and Express editions: Must be
an integer from 20 to 16384. * Provisioned IOPS storage (io1):
Enterprise and Standard editions: Must be an integer from 200
to 16384. Web and Express editions: Must be an integer from
100 to 16384. * Magnetic storage (standard): Enterprise and
Standard editions: Must be an integer from 200 to 1024. Web
and Express editions: Must be an integer from 20 to 1024.'
type: integer
allowMajorVersionUpgrade:
description: 'AllowMajorVersionUpgrade indicates that major version
upgrades are allowed. Changing this parameter doesn''t result
in an outage and the change is asynchronously applied as soon
as possible. Constraints: This parameter must be set to true
when specifying a value for the EngineVersion parameter that
is a different major version than the DB instance''s current
version.'
type: boolean
applyModificationsImmediately:
description: 'ApplyModificationsImmediately specifies whether
the modifications in this request and any pending modifications
are asynchronously applied as soon as possible, regardless of
the PreferredMaintenanceWindow setting for the DB instance.
If this parameter is set to false, changes to the DB instance
are applied during the next maintenance window. Some parameter
changes can cause an outage and are applied on the next call
to RebootDBInstance, or the next failure reboot. Review the
table of parameters in Modifying a DB Instance and Using the
Apply Immediately Parameter (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.Modifying.html)
in the Amazon RDS User Guide. to see the impact that setting
ApplyImmediately to true or false has for each modified parameter
and to determine when the changes are applied. Default: false'
type: boolean
autoMinorVersionUpgrade:
description: 'AutoMinorVersionUpgrade indicates that minor engine
upgrades are applied automatically to the DB instance during
the maintenance window. Default: true'
type: boolean
availabilityZone:
description: 'AvailabilityZone is the EC2 Availability Zone that
the DB instance is created in. For information on AWS Regions
and Availability Zones, see Regions and Availability Zones (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html).
Default: A random, system-chosen Availability Zone in the endpoint''s
AWS Region. Example: us-east-1d Constraint: The AvailabilityZone
parameter can''t be specified if the MultiAZ parameter is set
to true. The specified Availability Zone must be in the same
AWS Region as the current endpoint.'
type: string
backupRetentionPeriod:
description: 'BackupRetentionPeriod is the number of days for
which automated backups are retained. Setting this parameter
to a positive number enables backups. Setting this parameter
to 0 disables automated backups. Amazon Aurora Not applicable.
The retention period for automated backups is managed by the
DB cluster. For more information, see CreateDBCluster. Default:
1 Constraints: * Must be a value from 0 to 35 * Cannot be set
to 0 if the DB instance is a source to Read Replicas'
type: integer
caCertificateIdentifier:
description: CACertificateIdentifier indicates the certificate
that needs to be associated with the instance.
type: string
characterSetName:
description: CharacterSetName indicates that the DB instance should
be associated with the specified CharacterSet for supported
engines, Amazon Aurora Not applicable. The character set is
managed by the DB cluster. For more information, see CreateDBCluster.
type: string
cloudwatchLogsExportConfiguration:
description: CloudwatchLogsExportConfiguration is the configuration
setting for the log types to be enabled for export to CloudWatch
Logs for a specific DB instance.
properties:
disableLogTypes:
description: DisableLogTypes is the list of log types to disable.
items:
type: string
type: array
enableLogTypes:
description: EnableLogTypes is the list of log types to enable.
items:
type: string
type: array
type: object
copyTagsToSnapshot:
description: CopyTagsToSnapshot should be true to copy all tags
from the DB instance to snapshots of the DB instance, and otherwise
false. The default is false.
type: boolean
dbClusterIdentifier:
description: 'DBClusterIdentifier is the identifier of the DB
cluster that the instance will belong to. For information on
creating a DB cluster, see CreateDBCluster. Type: String'
type: string
dbInstanceClass:
description: DBInstanceClass is the compute and memory capacity
of the DB instance, for example, db.m4.large. Not all DB instance
classes are available in all AWS Regions, or for all database
engines. For the full list of DB instance classes, and availability
for your engine, see DB Instance Class (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html)
in the Amazon RDS User Guide.
type: string
dbName:
description: 'DBName is the meaning of this parameter differs
according to the database engine you use. Type: String MySQL
The name of the database to create when the DB instance is created.
If this parameter is not specified, no database is created in
the DB instance. Constraints: * Must contain 1 to 64 letters
or numbers. * Cannot be a word reserved by the specified database
engine MariaDB The name of the database to create when the DB
instance is created. If this parameter is not specified, no
database is created in the DB instance. Constraints: * Must
contain 1 to 64 letters or numbers. * Cannot be a word reserved
by the specified database engine PostgreSQL The name of the
database to create when the DB instance is created. If this
parameter is not specified, the default "postgres" database
is created in the DB instance. Constraints: * Must contain 1
to 63 letters, numbers, or underscores. * Must begin with a
letter or an underscore. Subsequent characters can be letters,
underscores, or digits (0-9). * Cannot be a word reserved by
the specified database engine Oracle The Oracle System ID (SID)
of the created DB instance. If you specify null, the default
value ORCL is used. You can''t specify the string NULL, or any
other reserved word, for DBName. Default: ORCL Constraints:
* Cannot be longer than 8 characters SQL Server Not applicable.
Must be null. Amazon Aurora The name of the database to create
when the primary instance of the DB cluster is created. If this
parameter is not specified, no database is created in the DB
instance. Constraints: * Must contain 1 to 64 letters or numbers.
* Cannot be a word reserved by the specified database engine'
type: string
dbParameterGroupName:
description: 'DBParameterGroupName is the name of the DB parameter
group to associate with this DB instance. If this argument is
omitted, the default DBParameterGroup for the specified engine
is used. Constraints: * Must be 1 to 255 letters, numbers, or
hyphens. * First character must be a letter * Cannot end with
a hyphen or contain two consecutive hyphens'
type: string
dbSecurityGroups:
description: 'DBSecurityGroups is a list of DB security groups
to associate with this DB instance. Default: The default DB
security group for the database engine.'
items:
type: string
type: array
dbSubnetGroupName:
description: DBSubnetGroupName is a DB subnet group to associate
with this DB instance. If there is no DB subnet group, then
it is a non-VPC DB instance.
type: string
dbSubnetGroupNameRef:
description: DBSubnetGroupNameRef is a reference to a DBSubnetGroup
used to set DBSubnetGroupName.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
dbSubnetGroupNameSelector:
description: DBSubnetGroupNameSelector selects a reference to
a DBSubnetGroup used to set DBSubnetGroupName.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
deletionProtection:
description: DeletionProtection indicates if the DB instance should
have deletion protection enabled. The database can't be deleted
when this value is set to true. The default is false. For more
information, see Deleting a DB Instance (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_DeleteInstance.html).
type: boolean
domain:
description: Domain specifies the Active Directory Domain to create
the instance in.
type: string
domainIAMRoleName:
description: DomainIAMRoleName specifies the name of the IAM role
to be used when making API calls to the Directory Service.
type: string
domainIAMRoleNameRef:
description: DomainIAMRoleNameRef is a reference to an IAMRole
used to set DomainIAMRoleName.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
domainIAMRoleNameSelector:
description: DomainIAMRoleNameSelector selects a reference to
an IAMRole used to set DomainIAMRoleName.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
enableCloudwatchLogsExports:
description: EnableCloudwatchLogsExports is the list of log types
that need to be enabled for exporting to CloudWatch Logs. The
values in the list depend on the DB engine being used. For more
information, see Publishing Database Logs to Amazon CloudWatch
Logs (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html#USER_LogAccess.Procedural.UploadtoCloudWatch)
in the Amazon Relational Database Service User Guide.
items:
type: string
type: array
enableIAMDatabaseAuthentication:
description: 'EnableIAMDatabaseAuthentication should be true to
enable mapping of AWS Identity and Access Management (IAM) accounts
to database accounts, and otherwise false. You can enable IAM
database authentication for the following database engines:
Amazon Aurora Not applicable. Mapping AWS IAM accounts to database
accounts is managed by the DB cluster. For more information,
see CreateDBCluster. MySQL * For MySQL 5.6, minor version 5.6.34
or higher * For MySQL 5.7, minor version 5.7.16 or higher Default:
false'
type: boolean
enablePerformanceInsights:
description: EnablePerformanceInsights should be true to enable
Performance Insights for the DB instance, and otherwise false.
For more information, see Using Amazon Performance Insights
(http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html)
in the Amazon Relational Database Service User Guide.
type: boolean
engine:
description: 'Engine is the name of the database engine to be
used for this instance. Not every database engine is available
for every AWS Region. Valid Values: * aurora (for MySQL 5.6-compatible
Aurora) * aurora-mysql (for MySQL 5.7-compatible Aurora) * aurora-postgresql
* mariadb * mysql * oracle-ee * oracle-se2 * oracle-se1 * oracle-se
* postgres * sqlserver-ee * sqlserver-se * sqlserver-ex * sqlserver-web
Engine is a required field'
type: string
engineVersion:
description: EngineVersion is the version number of the database
engine to use. For a list of valid engine versions, call DescribeDBEngineVersions.
The following are the database engines and links to information
about the major and minor versions that are available with Amazon
RDS. Not every database engine is available for every AWS Region.
Amazon Aurora Not applicable. The version number of the database
engine to be used by the DB instance is managed by the DB cluster.
For more information, see CreateDBCluster. MariaDB See MariaDB
on Amazon RDS Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html#MariaDB.Concepts.VersionMgmt)
in the Amazon RDS User Guide. Microsoft SQL Server See Version
and Feature Support on Amazon RDS (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.FeatureSupport)
in the Amazon RDS User Guide. MySQL See MySQL on Amazon RDS
Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.VersionMgmt)
in the Amazon RDS User Guide. Oracle See Oracle Database Engine
Release Notes (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.PatchComposition.html)
in the Amazon RDS User Guide. PostgreSQL See Supported PostgreSQL
Database Versions (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions)
in the Amazon RDS User Guide.
type: string
finalDBSnapshotIdentifier:
description: 'The DBSnapshotIdentifier of the new DBSnapshot created
when SkipFinalSnapshot is set to false. Specifying this parameter
and also setting the SkipFinalShapshot parameter to true results
in an error. Constraints: * Must be 1 to 255 letters or numbers.
* First character must be a letter * Cannot end with a hyphen
or contain two consecutive hyphens * Cannot be specified when
deleting a Read Replica.'
type: string
iops:
description: 'IOPS is the amount of Provisioned IOPS (input/output
operations per second) to be initially allocated for the DB
instance. For information about valid IOPS values, see see Amazon
RDS Provisioned IOPS Storage to Improve Performance (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#USER_PIOPS)
in the Amazon RDS User Guide. Constraints: Must be a multiple
between 1 and 50 of the storage amount for the DB instance.
Must also be an integer multiple of 1000. For example, if the
size of your DB instance is 500 GiB, then your IOPS value can
be 2000, 3000, 4000, or 5000.'
type: integer
kmsKeyId:
description: KMSKeyID for an encrypted DB instance. The KMS key
identifier is the Amazon Resource Name (ARN) for the KMS encryption
key. If you are creating a DB instance with the same AWS account
that owns the KMS encryption key used to encrypt the new DB
instance, then you can use the KMS key alias instead of the
ARN for the KM encryption key. Amazon Aurora Not applicable.
The KMS key identifier is managed by the DB cluster. For more
information, see CreateDBCluster. If the StorageEncrypted parameter
is true, and you do not specify a value for the KMSKeyID parameter,
then Amazon RDS will use your default encryption key. AWS KMS
creates the default encryption key for your AWS account. Your
AWS account has a different default encryption key for each
AWS Region.
type: string
licenseModel:
description: 'LicenseModel information for this DB instance. Valid
values: license-included | bring-your-own-license | general-public-license'
type: string
masterPasswordSecretRef:
description: MasterPasswordSecretRef references the secret that
contains the password used in the creation of this RDS instance.
If no reference is given, a password will be auto-generated.
properties:
key:
description: The key to select.
type: string
name:
description: Name of the secret.
type: string
namespace:
description: Namespace of the secret.
type: string
required:
- key
- name
- namespace
type: object
masterUsername:
description: 'MasterUsername is the name for the master user.
Amazon Aurora Not applicable. The name for the master user is
managed by the DB cluster. For more information, see CreateDBCluster.
MariaDB Constraints: * Required for MariaDB. * Must be 1 to
16 letters or numbers. * Cannot be a reserved word for the chosen
database engine. Microsoft SQL Server Constraints: * Required
for SQL Server. * Must be 1 to 128 letters or numbers. * The
first character must be a letter. * Cannot be a reserved word
for the chosen database engine. MySQL Constraints: * Required
for MySQL. * Must be 1 to 16 letters or numbers. * First character
must be a letter. * Cannot be a reserved word for the chosen
database engine. Oracle Constraints: * Required for Oracle.
* Must be 1 to 30 letters or numbers. * First character must
be a letter. * Cannot be a reserved word for the chosen database
engine. PostgreSQL Constraints: * Required for PostgreSQL. *
Must be 1 to 63 letters or numbers. * First character must be
a letter. * Cannot be a reserved word for the chosen database
engine.'
type: string
maxAllocatedStorage:
description: "The upper limit to which Amazon RDS can automatically
scale the storage of the DB instance. \n For more information
about this setting, including limitations that apply to it,
see Managing capacity automatically with Amazon RDS storage
autoscaling (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling)
in the Amazon RDS User Guide."
type: integer
monitoringInterval:
description: 'MonitoringInterval is the interval, in seconds,
between points when Enhanced Monitoring metrics are collected
for the DB instance. To disable collecting Enhanced Monitoring
metrics, specify 0. The default is 0. If MonitoringRoleARN is
specified, then you must also set MonitoringInterval to a value
other than 0. Valid Values: 0, 1, 5, 10, 15, 30, 60'
type: integer
monitoringRoleArn:
description: MonitoringRoleARN is the ARN for the IAM role that
permits RDS to send enhanced monitoring metrics to Amazon CloudWatch
Logs. For example, arn:aws:iam:123456789012:role/emaccess. For
information on creating a monitoring role, go to Setting Up
and Enabling Enhanced Monitoring (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html#USER_Monitoring.OS.Enabling)
in the Amazon RDS User Guide. If MonitoringInterval is set to
a value other than 0, then you must supply a MonitoringRoleARN
value.
type: string
monitoringRoleArnRef:
description: MonitoringRoleARNRef is a reference to an IAMRole
used to set MonitoringRoleARN.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
monitoringRoleArnSelector:
description: MonitoringRoleARNSelector selects a reference to
an IAMRole used to set MonitoringRoleARN.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
multiAZ:
description: MultiAZ specifies if the DB instance is a Multi-AZ
deployment. You can't set the AvailabilityZone parameter if
the MultiAZ parameter is set to true.
type: boolean
optionGroupName:
description: OptionGroupName indicates that the DB instance should
be associated with the specified option group. Permanent options,
such as the TDE option for Oracle Advanced Security TDE, can't
be removed from an option group, and that option group can't
be removed from a DB instance once it is associated with a DB
instance
type: string
performanceInsightsKMSKeyId:
description: PerformanceInsightsKMSKeyID is the AWS KMS key identifier
for encryption of Performance Insights data. The KMS key ID
is the Amazon Resource Name (ARN), KMS key identifier, or the
KMS key alias for the KMS encryption key.
type: string
performanceInsightsRetentionPeriod:
description: PerformanceInsightsRetentionPeriod is the amount
of time, in days, to retain Performance Insights data. Valid
values are 7 or 731 (2 years).
type: integer
port:
description: 'Port number on which the database accepts connections.
MySQL Default: 3306 Valid Values: 1150-65535 Type: Integer MariaDB
Default: 3306 Valid Values: 1150-65535 Type: Integer PostgreSQL
Default: 5432 Valid Values: 1150-65535 Type: Integer Oracle
Default: 1521 Valid Values: 1150-65535 SQL Server Default: 1433
Valid Values: 1150-65535 except for 1434, 3389, 47001, 49152,
and 49152 through 49156. Amazon Aurora Default: 3306 Valid Values:
1150-65535 Type: Integer'
type: integer
preferredBackupWindow:
description: 'PreferredBackupWindow is the daily time range during
which automated backups are created if automated backups are
enabled, using the BackupRetentionPeriod parameter. For more
information, see The Backup Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#USER_WorkingWithAutomatedBackups.BackupWindow)
in the Amazon RDS User Guide. Amazon Aurora Not applicable.
The daily time range for creating automated backups is managed
by the DB cluster. For more information, see CreateDBCluster.
The default is a 30-minute window selected at random from an
8-hour block of time for each AWS Region. To see the time blocks
available, see Adjusting the Preferred DB Instance Maintenance
Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#AdjustingTheMaintenanceWindow)
in the Amazon RDS User Guide. Constraints: * Must be in the
format hh24:mi-hh24:mi. * Must be in Universal Coordinated Time
(UTC). * Must not conflict with the preferred maintenance window.
* Must be at least 30 minutes.'
type: string
preferredMaintenanceWindow:
description: 'PreferredMaintenanceWindow is the time range each
week during which system maintenance can occur, in Universal
Coordinated Time (UTC). For more information, see Amazon RDS
Maintenance Window (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#Concepts.DBMaintenance).
Format: ddd:hh24:mi-ddd:hh24:mi The default is a 30-minute window
selected at random from an 8-hour block of time for each AWS
Region, occurring on a random day of the week. Valid Days: Mon,
Tue, Wed, Thu, Fri, Sat, Sun. Constraints: Minimum 30-minute
window.'
type: string
processorFeatures:
description: ProcessorFeatures is the number of CPU cores and
the number of threads per core for the DB instance class of
the DB instance.
items:
description: ProcessorFeature is a processor feature entry.
For more information, see Configuring the Processor of the
DB Instance Class (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html#USER_ConfigureProcessor)
in the Amazon RDS User Guide. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/ProcessorFeature
properties:
name:
description: Name of the processor feature. Valid names
are coreCount and threadsPerCore.
type: string
value:
description: Value of a processor feature name.
type: string
required:
- name
- value
type: object
type: array
promotionTier:
description: 'PromotionTier specifies the order in which an Aurora
Replica is promoted to the primary instance after a failure
of the existing primary instance. For more information, see Fault
Tolerance for an Aurora DB Cluster (http://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Backups.html#Aurora.Managing.FaultTolerance)
in the Amazon Aurora User Guide. Default: 1 Valid Values: 0
- 15'
type: integer
publiclyAccessible:
description: 'PubliclyAccessible specifies the accessibility options
for the DB instance. A value of true specifies an Internet-facing
instance with a publicly resolvable DNS name, which resolves
to a public IP address. A value of false specifies an internal
instance with a DNS name that resolves to a private IP address.
Default: The default behavior varies depending on whether DBSubnetGroupName
is specified. If DBSubnetGroupName is not specified, and PubliclyAccessible
is not specified, the following applies: * If the default VPC
in the target region doesn’t have an Internet gateway attached
to it, the DB instance is private. * If the default VPC in the
target region has an Internet gateway attached to it, the DB
instance is public. If DBSubnetGroupName is specified, and PubliclyAccessible
is not specified, the following applies: * If the subnets are
part of a VPC that doesn’t have an Internet gateway attached
to it, the DB instance is private. * If the subnets are part
of a VPC that has an Internet gateway attached to it, the DB
instance is public.'
type: boolean
region:
description: Region is the region you'd like your RDSInstance
to be created in.
type: string
restoreFrom:
description: RestoreFrom specifies the details of the backup to
restore when creating a new RDS instance. (If the RDS instance
already exists, this property will be ignored.)
properties:
s3:
description: S3 specifies the details of the S3 backup to
restore from.
properties:
bucketName:
description: BucketName is the name of the S3 bucket containing
the backup to restore.
type: string
bucketNameRef:
description: BucketNameRef is a reference to a Bucket
used to set BucketName.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
bucketNameSelector:
description: BucketNameSelector selects a reference to
a Bucket used to set BucketName.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching
labels is selected.
type: object
type: object
ingestionRoleARN:
description: IngestionRoleARN is the IAM role RDS can
assume that will allow it to access the contents of
the S3 bucket.
type: string
ingestionRoleARNRef:
description: IngestionRoleARNRef is a reference to a IAM
Role used to set IngestionRoleARN.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
ingestionRoleARNSelector:
description: IngestionRoleARNSelector selects a reference
to a IAM Role used to set IngestionRoleARN.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object
with the same controller reference as the selecting
object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching
labels is selected.
type: object
type: object
prefix:
description: Prefix is the path prefix of the S3 bucket
within which the backup to restore is located.
type: string
sourceEngine:
description: SourceEngine is the engine used to create
the backup. Must be "mysql".
type: string
sourceEngineVersion:
description: 'SourceEngineVersion is the version of the
engine used to create the backup. Example: "5.7.30"'
type: string
required:
- sourceEngine
- sourceEngineVersion
type: object
snapshot:
description: Snapshot specifies the details of the database
snapshot to restore from.
properties:
snapshotIdentifier:
description: SnapshotIdentifier is the identifier of the
database snapshot to restore.
type: string
required:
- snapshotIdentifier
type: object
source:
description: Source is the type of the backup to restore when
creating a new RDS instance. Only S3 and Snapshot are supported
at present.
enum:
- S3
- Snapshot
type: string
required:
- source
type: object
scalingConfiguration:
description: ScalingConfiguration is the scaling properties of
the DB cluster. You can only modify scaling properties for DB
clusters in serverless DB engine mode.
properties:
autoPause:
description: AutoPause specifies whether to allow or disallow
automatic pause for an Aurora DB cluster in serverless DB
engine mode. A DB cluster can be paused only when it's idle
(it has no connections). If a DB cluster is paused for more
than seven days, the DB cluster might be backed up with
a snapshot. In this case, the DB cluster is restored when
there is a request to connect to it.
type: boolean
maxCapacity:
description: MaxCapacity is the maximum capacity for an Aurora
DB cluster in serverless DB engine mode. Valid capacity
values are 2, 4, 8, 16, 32, 64, 128, and 256. The maximum
capacity must be greater than or equal to the minimum capacity.
type: integer
minCapacity:
description: MinCapacity is the minimum capacity for an Aurora
DB cluster in serverless DB engine mode. Valid capacity
values are 2, 4, 8, 16, 32, 64, 128, and 256. The minimum
capacity must be less than or equal to the maximum capacity.
type: integer
secondsUntilAutoPause:
description: SecondsUntilAutoPause is the time, in seconds,
before an Aurora DB cluster in serverless mode is paused.
type: integer
type: object
skipFinalSnapshotBeforeDeletion:
description: 'Determines whether a final DB snapshot is created
before the DB instance is deleted. If true is specified, no
DBSnapshot is created. If false is specified, a DB snapshot
is created before the DB instance is deleted. Note that when
a DB instance is in a failure state and has a status of ''failed'',
''incompatible-restore'', or ''incompatible-network'', it can
only be deleted when the SkipFinalSnapshotBeforeDeletion parameter
is set to "true". Specify true when deleting a Read Replica.
The FinalDBSnapshotIdentifier parameter must be specified if
SkipFinalSnapshotBeforeDeletion is false. Default: false'
type: boolean
storageEncrypted:
description: 'StorageEncrypted specifies whether the DB instance
is encrypted. Amazon Aurora Not applicable. The encryption for
DB instances is managed by the DB cluster. For more information,
see CreateDBCluster. Default: false'
type: boolean
storageType:
description: 'StorageType specifies the storage type to be associated
with the DB instance. Valid values: standard | gp2 | io1 If
you specify io1, you must also include a value for the IOPS
parameter. Default: io1 if the IOPS parameter is specified,
otherwise standard'
type: string
tags:
description: Tags. For more information, see Tagging Amazon RDS
Resources (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html)
in the Amazon RDS User Guide.
items:
description: Tag is a metadata assigned to an Amazon RDS resource
consisting of a key-value pair. Please also see https://docs.aws.amazon.com/goto/WebAPI/rds-2014-10-31/Tag
properties:
key:
description: 'A key is the required name of the tag. The
string value can be from 1 to 128 Unicode characters in
length and can''t be prefixed with "aws:" or "rds:". The
string can only contain only the set of Unicode letters,
digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'',
''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").'
type: string
value:
description: 'A value is the optional value of the tag.
The string value can be from 1 to 256 Unicode characters
in length and can''t be prefixed with "aws:" or "rds:".
The string can only contain only the set of Unicode letters,
digits, white-space, ''_'', ''.'', ''/'', ''='', ''+'',
''-'' (Java regex: "^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$").'
type: string
type: object
type: array
timezone:
description: Timezone of the DB instance. The time zone parameter
is currently supported only by Microsoft SQL Server (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.TimeZone).
type: string
useDefaultProcessorFeatures:
description: A value that specifies that the DB instance class
of the DB instance uses its default processor features.
type: boolean
vpcSecurityGroupIDRefs:
description: VPCSecurityGroupIDRefs are references to VPCSecurityGroups
used to set the VPCSecurityGroupIDs.
items:
description: A Reference to a named object.
properties:
name:
description: Name of the referenced object.
type: string
required:
- name
type: object
type: array
vpcSecurityGroupIDSelector:
description: VPCSecurityGroupIDSelector selects references to
VPCSecurityGroups used to set the VPCSecurityGroupIDs.
properties:
matchControllerRef:
description: MatchControllerRef ensures an object with the
same controller reference as the selecting object is selected.
type: boolean
matchLabels:
additionalProperties:
type: string
description: MatchLabels ensures an object with matching labels
is selected.
type: object
type: object
vpcSecurityGroupIds:
description: 'VPCSecurityGroupIDs is a list of EC2 VPC security
groups to associate with this DB instance. Amazon Aurora Not
applicable. The associated list of EC2 VPC security groups is
managed by the DB cluster. For more information, see CreateDBCluster.
Default: The default EC2 VPC security group for the DB subnet
group''s VPC.'
items:
type: string
type: array
required:
- dbSubnetGroup
- securityGroup
- dbParameterGroup
- rdsInstance
required:
- parameters
status:
type: object
properties:
databaseInstanceName:
description: Name of the bucket
type: string
components:
type: object
properties:
# -------------------------------------------------------------
dbSubnetGroup:
description: Conditions of the resource.
items:
description: A Condition that may apply to a resource.
properties:
lastTransitionTime:
description: LastTransitionTime is the last time this condition
transitioned from one status to another.
format: date-time
type: string
message:
description: A Message containing details about this condition's
last transition from one status to another, if any.
type: string
reason:
description: A Reason for this condition's last transition from
one status to another.
type: string
status:
description: Status of this condition; is it currently True,
False, or Unknown?
type: string
type:
description: Type of this condition. At most one of each condition
type may apply to a resource at any point in time.
type: string
required:
- lastTransitionTime
- reason
- status
- type
type: object
type: array
# -------------------------------------------------------------
securityGroup:
description: Conditions of the resource.
items:
description: A Condition that may apply to a resource.
properties:
lastTransitionTime:
description: LastTransitionTime is the last time this condition
transitioned from one status to another.
format: date-time
type: string
message:
description: A Message containing details about this condition's
last transition from one status to another, if any.
type: string
reason:
description: A Reason for this condition's last transition from
one status to another.
type: string
status:
description: Status of this condition; is it currently True,
False, or Unknown?
type: string
type:
description: Type of this condition. At most one of each condition
type may apply to a resource at any point in time.
type: string
required:
- lastTransitionTime
- reason
- status
- type
type: object
type: array
# -------------------------------------------------------------
dbParameterGroups:
description: Conditions of the resource.
items:
description: A Condition that may apply to a resource.
properties:
lastTransitionTime:
description: LastTransitionTime is the last time this condition
transitioned from one status to another.
format: date-time
type: string
message:
description: A Message containing details about this condition's
last transition from one status to another, if any.
type: string
reason:
description: A Reason for this condition's last transition from
one status to another.
type: string
status:
description: Status of this condition; is it currently True,
False, or Unknown?
type: string
type:
description: Type of this condition. At most one of each condition
type may apply to a resource at any point in time.
type: string
required:
- lastTransitionTime
- reason
- status
- type
type: object
type: array
# -------------------------------------------------------------
rdsInstance:
description: Conditions of the resource.
items:
description: A Condition that may apply to a resource.
properties:
lastTransitionTime:
description: LastTransitionTime is the last time this condition
transitioned from one status to another.
format: date-time
type: string
message:
description: A Message containing details about this condition's
last transition from one status to another, if any.
type: string
reason:
description: A Reason for this condition's last transition from
one status to another.
type: string
status:
description: Status of this condition; is it currently True,
False, or Unknown?
type: string
type:
description: Type of this condition. At most one of each condition
type may apply to a resource at any point in time.
type: string
required:
- lastTransitionTime
- reason
- status
- type
type: object
type: array
While checking our setup I found this after describing an instance, seems like the kind keeps being rewritten and that may be the trigger for all of our issues.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BindCompositeResource 29m (x6808 over 20d) offered/compositeresourcedefinition.apiextensions.crossplane.io Successfully bound composite resource
Normal ConfigureCompositeResource 6m59s (x6834 over 20d) offered/compositeresourcedefinition.apiextensions.crossplane.io Successfully applied composite resource
And all of our objetcs are impacted, the number increase each time I run the command.
$ k get mysql,pgsql -A -o=custom-columns='RESOURCE_VERSION:.metadata.resourceVersion'
RESOURCE_VERSION
92920052
92920175
92919353
92919299
[...]
Could you also take a look into what kind of resource is tagged with AddTagsToResource?
I gathered all events from the last 10 hours and the targets are only the RDS DB object, it doesnt impact any other object (parameter group, security group ...).
However, I found out that not all the instances are on the list, and I found why, the instances with an empty rdsInstance.forProvider.dbName are the ones being Tagged over and over.
That's something we already made mandatory on our latests templates, people may be using an outdated version and we are going to fix this ASAP.
k get mysql -A -o=custom-columns='UID:.metadata.uid,DBNAME:.spec.parameters.rdsInstance.forProvider.dbName'
$ k get mysql -A -o=custom-columns='UID:.metadata.uid,DBNAME:.spec.parameters.rdsInstance.forProvider.dbName'
UID DBNAME
bdf0324d-9900-4181-b5f4-cd689719b917 redacted
6151c630-d994-4b5d-8d5b-0e2eeacd5ca7 redacted
e8329191-9c5e-4598-91f9-fdc04917691e redacted
c8990313-b451-4316-829d-d7d308abf410 redacted
6d678206-4311-453f-884a-f1be7eb00948 redacted
71fd8bdc-4b21-40bf-9bc1-3b65a7b8a8b1 redacted
dde38a53-cb1f-45a4-9322-cf78afb6f0b8 redacted
b268c13f-a8b8-47cf-bd84-9a572db81cfc
9185f438-9328-4c37-a095-aba5ca8c7805
807f978a-efe5-43e9-859a-70151b1bd9c4
995a9674-8348-4b34-a1a4-6a07e006ff0f
63917556-a50c-43dc-93a5-b1e725beace7
951137cb-3c5c-4acc-87b6-6e4bd66aa222
6d925516-8510-43ac-bc06-08233cca36e8
5166ef3c-4c30-4240-8e7a-3cedef03caf9
26114338-a911-43d0-afdf-a204cc8dac65
a1fe93b4-967e-4c9a-801c-2b3e8ffbe6ed
738ae87e-f9d2-412f-a9eb-b25b8a6c8756
25847b29-914a-4221-a0de-e5d9050d1fac redacted
fbf78898-572c-4b5e-9cc8-28d11833d54b redacted
2595e93e-dfea-4821-930d-7b865d6216ab
b43c9e03-c4c1-4947-ac00-ea12b27fdd66
1761b0ba-b78f-4022-b78d-a615fd9d672e redacted
22997ec5-c95d-451c-ade0-2c6c353a2fe4 redacted
ae7cc3bf-a06a-489d-b98d-c8181591e30a redacted
545e7dd7-9148-43de-8892-f24c78fbe31f redacted
I wonder if you could upgrade to 0.29 to check if the situation has improved.
We need to test the impacts of a live upgrade on another cluster, could be during this week.
UpperCase in maintenance/backup window
We do not specify the values, it uses the defaults
Thanks again for your help
Hello @chlunde ,
The situation is worsening, we are being throttled by AWS due to the calls increasing, and we requested a quota extension for this, however it's difficult to explain to AWS why there is nearly 50 calls per second for only 30 databases deployed.
'AddTagsToResource': 435,
'DescribeDBClusterParameterGroups': 17042,
'DescribeDBClusterParameters': 7925,
'DescribeDBClusters': 18020,
'DescribeDBInstances': 49925,
'DescribeDBParameterGroups': 16505,
'DescribeDBParameters': 15496,
'DescribeDBSubnetGroups': 28931,
'ListTagsForResource': 19208,
'ModifyDBInstance': 558,
'ModifyDBParameterGroup': 67
'Processed Events': 174112,
'Timeframe': '- 60 minutes'
We reduced the number of databases without rdsInstance.forProvider.dbName, not sure why it's related, but AddTagsToResource has decreased.
We also are planning to upgrade to 0.29.
EDIT : We suspect something, the patches in the compositions that check and posts the status from each component :
- type: ToCompositeFieldPath
fromFieldPath: status.conditions
toFieldPath: status.components.dbSubnetGroup
We are going to test to disable it, and if it does not break anything, deploy it in production and re-assess the calls numbers.
Do we have another way to mitigate those issues ?
Update : after some back-and-forth with AWS and a lot of digging from us, we found out that when the provider has an issue to sync an object, it keeps retrying endlessly.
As the calls are retrying, more and more calls are being throttled and it hits the rate limit and is stuck forever. AWS increased temporarily our rate limit and the provider managed to sync, then the calls dropped to a more normal rate.
From AWS :
It is important to highlight that the call rate dropped to 1/3 (from 50 req/s to 16 req/s) after the update,
this is due to the fact that you are constantly retrying when you get throttled.
Suggested to review the automation process to avoid those retry storms.
Those limits will be removed in two weeks from now, unless you have a valid business case to keep them.
Regarding our setup, we upgraded to 0.29.0 - no changes, we removed the components status - no changes. We are still wondering how to cap the calls number and -if possible- enable a ExponentialBackOff in order to avoid the situation that may happen again.
anyone was able to fix it? we are having the same problem too many api calls resulting in throttling, specially for cloudfront crossplane version: 1.10 aws-provider: 0.32 eks: 1.22
Thanks
The changes in https://github.com/crossplane-contrib/provider-aws/pull/1705 may have solved much of this issue - they are in the latest release - 0.39.0
Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}