aws-airflow-stack
aws-airflow-stack copied to clipboard
villasv
Install extra packages in the example project
A common requirement. Also relevant to show that #172 is not a reason to fork the project.
Example project can be found here:
https://github.com/villasv/aws-airflow-stack/tree/master/examples/project
This is what I'm working from at present, I've added 3 components to attempt to install python dependencies onto the workers during autoscaling
- This block to the appspec:
hooks: BeforeInstall:
- location: scripts/install_dependencies.sh timeout: 300 runas: root
- This install script:
#!/bin/bash pip3 install -r ../requirements.txt exit 0
- This requirements file
pip-install-test==0.5
Unfortunately, I'm getting errors in the logs around the install_dependencies.sh script and I'm not sure why (this is my first exposure to cloudformation & codedeploy so I hope I'm overlooking something simple)
On Tue, Apr 28, 2020 at 9:11 AM Victor Villas [email protected] wrote:
Example project can be found here:
https://github.com/villasv/aws-airflow-stack/tree/master/examples/project
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/177#issuecomment-620704533, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPOTP3YHOJP4SSVLEYDRO352LANCNFSM4MSZSVLA .
Your appspec loogs good, your requirements file too.
The one thing yours is different from what I have in the development branch (https://github.com/villasv/aws-airflow-stack/tree/develop/examples/project/airflow) is that you're providing a relative path to the requirements file.
Also, you might want to use AfterInstall instead of BeforeInstall to make sure the requirements file is already in place.
Omg, if this solves it I am going to be simultaneously super relived and infuriated (I think I tried every combination of paths I could reasonably think of except apparently this.)
On Tue, Apr 28, 2020 at 9:42 AM Victor Villas [email protected] wrote:
Your appspec loogs good, your requirements file too.
The one thing yours is different from what I have in the development branch ( https://github.com/villasv/aws-airflow-stack/tree/develop/examples/project/airflow) is that you're providing a relative path to the requirements file.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/177#issuecomment-620723220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPPPIQJPBPDVRQVTVBLRO4BORANCNFSM4MSZSVLA .