Avoid reconciling many objects on the same database at the same time

[chlunde]

What problem are you facing?

For smaller RDS instances, we see CPU spikes when many provider-sql objects are reconciled at the same time. This can affect applications using the same database

How could Crossplane help solve your problem?

Avoid reconciling all objects in a burst. For example:

• Connection reuse/pooling
• Max N objects per hostname reconciled at the same time
• Add jitter to spread reconciles
• Add poll interval flag

https://pkg.go.dev/database/sql#Open

The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.

There's also: https://pkg.go.dev/database/sql#DB.SetMaxOpenConns