Race condition when deleting users

[mhutter]

What happened?

It might happen that a race-condition in crossplane might lead to a reconcileError, because the secret has already been deleted. This can easily be observed by looking at the affected user object. If the object has a field Deletion Timestamp as well as message stating that the password secret can't be found, then you need to edit the affected user object and remove the finalizers field. Once the field has been removed, the user object should be deleted automatically.

How can we reproduce it?

Create a Database and Users via Crossplane. Delete everything

What environment did it happen in?

Crossplane version: crossplane:v1.1.1 Kubernetes Version: Server Version: v1.19.9+vmware.1

[sendmars]

I'm seeing this too. I have a composition for AuroraPostgres in AWS with base resources:

Using provider-aws:

• DBSubnetGroup, database.aws.crossplane.io/v1beta1
• SecurityGroup, ec2.aws.crossplane.io/v1beta1
• DBCluster, rds.aws.crossplane.io/v1alpha1
• DBInstance (R+W instances), rds.aws.crossplane.io/v1alpha1

Using provider-sql:

• ProviderConfig, postgresql.sql.crossplane.io/v1alpha1
• Role, postgresql.sql.crossplane.io/v1alpha1
• Grant, postgresql.sql.crossplane.io/v1alpha1

All resources are correctly created, ready and synced.

When deleting, I issue a request to delete the claim: kubectl delete aurorapostgres foo-1.

The role is the one that usually doesn't get deleted, and it's keeping its finalizer. I suspect that there's a race condition and that it tries to connect to the database that has already been deleted in order to delete the role, which then never happens.

[mateusz-lubanski-sinch]

I observed same behaviour for:

• RDSInstance provisioned via provider-aws
• resources provisioned via provider-sql (e.g. mysql.sql.crossplane.io/v1alpha1/User, mysql.sql.crossplane.io/v1alpha1/Grant, mysql.sql.crossplane.io/v1alpha1/Database)

If RDSInstance is deleted before resources provisioned via provider-sql than I get same error

Warning  CannotConnectToProvider  6m (x59 over 42h)  managed/database.mysql.sql.crossplane.io  cannot get credentials Secret: Secret "099fb601-3c62-4ea8-8c26-595c0773f2eb-rdsinstance" not found

This is problematic for us as we are using FLUX to deploy/delete our k8s objects and we don't have a control over an order in which kubernetes/flux will remove crossplane resources

[na4ma4]

I think there is three separate errors here but linked to a common issue, we're seeing the same thing with postgresql and provider-sql.

I'm not sure why the secret is required to delete the roles though, the provider has superuser access, it created the roles and can delete them as well.

[Duologic]

Closing as stale. Please re-open and update if this is still valid.