Why not choose Aurora Serverless instead of RDS?

[readybuilderone]

It's cool to run the workload on Serverless Fargate, but why not use Aurora Serverless to replace the RDS? so the total solution would be serverless, is that a good idea?

[thechaithanya]

That's a valid point. We too considered adding Aurora Serverless as default offering. Aurora Serverless is useful when the traffic is low (dev/beta environments) or unpredictable. For continuous/steady/huge traffic, it will be costlier compared to RDS (reserved RDS will be much cheaper)

Metadata DB will have regular/continuous traffic in the form of heartbeat and other DAG/Task execution data. So, we went ahead in favor of RDS (a smaller, free-tier instance)

Being said that, it is easy to swap out RDS with Aurora Serverless. We have to replace this block of code with something like this Feel free to submit a PR, maybe a new branch, if you end up using Aurora Serverless instead of RDS.

[readybuilderone]

Thanks @thechaithanya for your suggestion. I am a solution architecture in AWS from China, since Managed Airflow hasn't been launched in China yet, the solution is extremely important, I'd like to submit a PR to replace RDS using Aurora Serverless, thanks again.

[cloneluke]

@readybuilderone you should make it an option, to choose aurora or RDS, not eliminate RDS

[cloneluke]

Yes, I will eventually tackle this as well but its lower down my priority list for this repo