terraform-aws-airflow
terraform-aws-airflow copied to clipboard
unruly
Terraform module for a PostgreSQL-backed Apache Airflow instance
terraform-aws-airflow
Terraform module to deploy an Apache Airflow instance on AWS backed by an RDS PostgreSQL database for storage.
How to use this Module
You can use this module from the Terraform Registry
module "example-airflow-setup" {
source = "unruly/airflow/aws"
key = "your-aws-keypair"
db_password = "some-db-password"
fernet_key = "your-fernet-key"
vpc_id = "some-vpc-id"
security_group_id = "some-security-group-id"
subnet_ids = [ "id-1", "id-2" ]
}
Configuration
| Argument | Description |
|---|---|
| ami | AMI code for the Airflow server (default: ami-a042f4d8 - CentOS 7 community image) |
| instance_type | Instance type for the Airflow server (default: c3.xlarge) |
| key | AWS SSH Key Pair name |
| subnet_ids | List of AWS subnet ids for Airflow server and database |
| vpc_id | AWS VPC in which to create the Airflow server |
| security_group_id | AWS Security group in which to create the Airflow server |
| db_password | Password for the PostgreSQL instance |
| fernet_key | Key for encrypting data in the database - see Airflow docs |
Outputs
| Output | Description |
|---|---|
| airflow_instance_public_dns | Public DNS for the Airflow instance |
| airflow_instance_public_ip | Public IP address for the Airflow instance |
| airflow_instance_private_ip | Private IP for the Airflow instance |
| airflow_database_security_group_id | Security group id for the Airflow database |
Quick-start
In the ./sample directory, there is a terraform configuration file (main.tf) and an Airflow DAG file (example-dag.py).
-
Set up Airflow in AWS eu-west-1
terraform apply -var "key_name=<YOUR-AWS-KEYPAIR-NAME>" -
Wait until the webserver has started - get the url for Airflow with
terraform output airflow_public_dnsand navigate to it using your browser or curl -
Copy across the example DAG to the instance
scp -o StrictHostKeyChecking=no example-dag.py centos@$(terraform output airflow_public_dns):/home/centos/airflow/dags -
Wait for Airflow to pick up the DAG and display it on the page as below
-
Congratulations, you have a running Airflow server in production!
WARNING - the database passwords and Fernet key are hardcoded in the sample configuration: do not use these in production!
unruly