docker-airflow
docker-airflow copied to clipboard
puckel
Unable to receive email on task failure or even using EmailOperator
Hi Guys,
I am unable to receive email from my box even after adding required parameters to send one.
Below is how my default args looks like --
default_args = {
'owner': 'phonrao',
'depends_on_past': False,
#'start_date': datetime(2019, 3, 28),
'start_date': airflow.utils.dates.days_ago(2),
'email': ['[email protected]'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 1,
'retry_delay': timedelta(minutes=5),
#'on_failure_callback': report_failure,
#'end_date': datetime(2020,4 ,1),
#'schedule_interval': '@hourly',
}
I have few HttpsOperator task in between -- those are working good and are a success, but they do not send email on error(I purposely tried to introduce an error to check if they send any email). Below is an example of my task.
t1 = SimpleHttpOperator(
task_id='t1',
http_conn_id='http_waterfall',
endpoint='/update_data',
method='POST',
headers={"Content-Type":"application/json"},
xcom_push=True,
log_response=True,
dag=dag,
)
and this is my EmailOperator task
t2 = EmailOperator(
dag=dag,
task_id="send_email",
to='[email protected]',
subject='Success',
html_content="<h3>Success</h3>"
)
t2 >> t1
Below is the error from Logs:
[2019-04-02 15:28:21,305] {{base_task_runner.py:101}} INFO - Job 845: Subtask send_email [2019-04-02 15:28:21,305] {{cli.py:520}} INFO - Running <TaskInstance: schedulerDAG.send_email 2019-04-02T15:23:08.896589+00:00 [running]> on host a47cd79aa987
[2019-04-02 15:28:21,343] {{logging_mixin.py:95}} INFO - [2019-04-02 15:28:21,343] {{configuration.py:255}} WARNING - section/key [smtp/smtp_user] not found in config
[2019-04-02 15:28:21,343] {{models.py:1788}} ERROR - [Errno 99] Cannot assign requested address
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1657, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/operators/email_operator.py", line 78, in execute
mime_subtype=self.mime_subtype, mime_charset=self.mime_charset)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/email.py", line 55, in send_email
mime_subtype=mime_subtype, mime_charset=mime_charset, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/email.py", line 101, in send_email_smtp
send_MIME_email(smtp_mail_from, recipients, msg, dryrun)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/email.py", line 121, in send_MIME_email
s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
File "/usr/local/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/local/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/local/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
[2019-04-02 15:28:21,351] {{models.py:1817}} INFO - All retries failed; marking task as FAILED
Has anyone encounter this issue and any suggestions on how do I resolve this?
This is a problem of the Dockerfile, on line 74 the config/airflow.cfg is copied to ${AIRFLOW_HOME}/airflow.cfg:
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
And all the docker uses the other config file ./airflow.cfg, a fast solution is just to copy your airflow.cfg to the config/airflow.cfg that will make the worker have the right configs that should solve the problem.
Another way to solve the problem is change the line 74 of the Dockerfile to:
COPY airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
To add more context, I use gmail settings from this post: https://stackoverflow.com/questions/51829200/how-to-set-up-airflow-send-email and it works fine when deployed on bare metal. When deployed in Kubernetes, the email setup seems to fail.
Here is the error message I get for using the email operator: [2019-10-16 09:11:14,149] {logging_mixin.py:95} INFO - [[34m2019-10-16 09:11:14,148[0m] {[34mconfiguration.py:[0m293} WARNING[0m - section/key [[1msmtp[0m/[1msmtp_user[0m] not found in config[0m [2019-10-16 09:11:14,149] {taskinstance.py:1047} ERROR - [Errno 99] Cannot assign requested address Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 922, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.7/site-packages/airflow/operators/email_operator.py", line 78, in execute mime_subtype=self.mime_subtype, mime_charset=self.mime_charset) File "/usr/local/lib/python3.7/site-packages/airflow/utils/email.py", line 55, in send_email mime_subtype=mime_subtype, mime_charset=mime_charset, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/utils/email.py", line 101, in send_email_smtp send_MIME_email(smtp_mail_from, recipients, msg, dryrun) File "/usr/local/lib/python3.7/site-packages/airflow/utils/email.py", line 121, in send_MIME_email s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT) File "/usr/local/lib/python3.7/smtplib.py", line 251, in init (code, msg) = self.connect(host, port) File "/usr/local/lib/python3.7/smtplib.py", line 336, in connect self.sock = self._get_socket(host, port, self.timeout) File "/usr/local/lib/python3.7/smtplib.py", line 307, in _get_socket self.source_address) File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection raise err File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection sock.connect(sa) OSError: [Errno 99] Cannot assign requested address
Hi,
Any found solution for this error..??? Please help community members to resolve this issue...
hi, any solution for this problem?
because i'm getting this error too
currently use airflow version 2.0 and docker version 3
