airflow-on-kubernetes
airflow-on-kubernetes copied to clipboard
Published
20 hours ago •
stwind
stwind
Webserver Installation issue
Seems airflow installation get stuck while running below command.
I see the airflow pod is in error and shows only below
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/airflow to minikube
Normal Pulled 6s kubelet, minikube Container image "anayyar82/airflow-aks" already present on machine
Normal Created 6s kubelet, minikube Created container webserver
Normal Started 5s kubelet, minikube Started container webserver
Normal Pulled 5s kubelet, minikube Container image "anayyar82/airflow-aks" already present on machine
Normal Created 5s kubelet, minikube Created container scheduler
Normal Started 5s kubelet, minikube Started container scheduler
(base) C02YR2K0LVDQ:airflow-on-kubernetes ankur.nayyar$ kubectl get pod airflow
running the code:
{
"spec": {
"containers":[{
"name": "webserver",
"image": "my/airflow",
"imagePullPolicy":"IfNotPresent",
"command": ["airflow","webserver"],
"stdin": true,
"tty": true,
"env": [
{"name":"AIRFLOW__CORE__LOAD_EXAMPLES","value":"False"},
{"name":"AIRFLOW__CORE__SQL_ALCHEMY_CONN","value":"mysql://airflow:[email protected]/airflow"},
{"name":"AIRFLOW__CORE__EXECUTOR","value":"KubernetesExecutor"},
{"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY","value":"my/airflow"},
{"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG","value":"latest"},
{"name":"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST","value":"'$PWD/dags'"}
],
"volumeMounts": [{"mountPath": "/var/lib/airflow/dags","name": "store"}]
},{
"name": "scheduler",
"image": "my/airflow",
"imagePullPolicy":"IfNotPresent",
"command": ["airflow","scheduler"],
"stdin": true,
"tty": true,
"env": [
{"name":"AIRFLOW__CORE__LOAD_EXAMPLES","value":"False"},
{"name":"AIRFLOW__CORE__SQL_ALCHEMY_CONN","value":"mysql://airflow:[email protected]/airflow"},
{"name":"AIRFLOW__CORE__EXECUTOR","value":"KubernetesExecutor"},
{"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY","value":"my/airflow"},
{"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG","value":"latest"},
{"name":"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST","value":"'$PWD/dags'"},
{"name":"AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS","value":""}
],
"volumeMounts": [{"mountPath": "/var/lib/airflow/dags","name": "store"}]
}],
"volumes": [{"name":"store","hostPath":{"path":"'$PWD/dags'","type":"Directory"}}]
}
}'```
Hi @anayyar82 , you can use kubectl logs to see detail logs inside the container.
Thanks @stwind Yeah i did but seems it stucked somehow.
kubectl logs -f airflow -c scheduler
[2020-04-06 14:33:31,551] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-04-06 14:33:33,244] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2020-04-06 14:33:33,303] {scheduler_job.py:1344} INFO - Starting the scheduler
[2020-04-06 14:33:33,303] {scheduler_job.py:1352} INFO - Running execute loop for -1 seconds
[2020-04-06 14:33:33,305] {scheduler_job.py:1353} INFO - Processing each file at most -1 times
[2020-04-06 14:33:33,306] {scheduler_job.py:1356} INFO - Searching for files in /tmp/airflow/dags
[2020-04-06 14:33:33,308] {scheduler_job.py:1358} INFO - There are 0 files in /tmp/airflow/dags
[2020-04-06 14:33:33,317] {kubernetes_executor.py:805} INFO - Start Kubernetes executor
[2020-04-06 14:33:53,879] {kubernetes_executor.py:351} INFO - Event: and now my watch begins starting at resource_version: 0
[2020-04-06 14:33:36,929] {kubernetes_executor.py:733} INFO - When executor started up, found 0 queued task instances
[2020-04-06 14:33:53,933] {scheduler_job.py:1409} INFO - Resetting orphaned tasks for active dag runs
[2020-04-06 14:33:54,212] {dag_processing.py:556} INFO - Launched DagFileProcessorManager with pid: 29
[2020-04-06 14:33:54,664] {settings.py:54} INFO - Configured default timezone <Timezone [UTC]>
[2020-04-06 14:33:54,693] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=29
and for Webserver
kubectl logs -f airflow -c webserver
[2020-04-06 14:33:31,551] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
May be this helps
kubectl describe pod airflow
Name: airflow
Namespace: default
Priority: 0
Node: minikube/192.168.64.2
Start Time: Mon, 06 Apr 2020 09:33:11 -0500
Labels: run=airflow
Annotations: <none>
Status: Running
IP: 172.17.0.2
IPs:
IP: 172.17.0.2
Containers:
webserver:
Container ID: docker://f09249e339079acd78e2175ef690a853dd264664bf53ac826f743c46e83bc7b4
Image: anayyar82/airflow-aks1
Image ID: docker-pullable://anayyar82/airflow-aks1@sha256:7f459e5404ab93668ed3a9476ac02cda8acc691aae1b8b983e65611109c546be
Port: <none>
Host Port: <none>
Command:
airflow
webserver
State: Terminated
Reason: Error
Exit Code: 137
Started: Mon, 06 Apr 2020 09:33:30 -0500
Finished: Mon, 06 Apr 2020 09:33:53 -0500
Ready: False
Restart Count: 0
Environment:
AIRFLOW__CORE__LOAD_EXAMPLES: False
AIRFLOW__CORE__SQL_ALCHEMY_CONN: mysql://airflow:[email protected]/airflow
AIRFLOW__CORE__EXECUTOR: KubernetesExecutor
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: anayyar82/airflow-aks1
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: latest
AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST: /tmp/airflow/dags
Mounts:
/tmp/airflow/dags from store (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-n8qt9 (ro)
scheduler:
Container ID: docker://8a6731a2e07caa9e0f58cbc26a94e7c897fc3354e2d8e593461db946c8fba763
Image: anayyar82/airflow-aks1
Image ID: docker-pullable://anayyar82/airflow-aks1@sha256:7f459e5404ab93668ed3a9476ac02cda8acc691aae1b8b983e65611109c546be
Port: <none>
Host Port: <none>
Command:
airflow
scheduler
State: Running
Started: Mon, 06 Apr 2020 09:33:30 -0500
Ready: True
Restart Count: 0
Environment:
AIRFLOW__CORE__LOAD_EXAMPLES: False
AIRFLOW__CORE__SQL_ALCHEMY_CONN: mysql://airflow:[email protected]/airflow
AIRFLOW__CORE__EXECUTOR: KubernetesExecutor
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: anayyar82/airflow-aks1
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: latest
AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST: /tmp/airflow/dags
AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS:
Mounts:
/tmp/airflow/dags from store (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-n8qt9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
store:
Type: HostPath (bare host directory volume)
Path: /tmp/airflow/dags
HostPathType:
default-token-n8qt9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-n8qt9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
sorry by just looking at the logs I am not quite sure what is the problem, maybe you could try running without k8s first?
Yeah without it's running fine. Could you please recheck if something is missing in the doc?