astronomer-providers icon indicating copy to clipboard operation
astronomer-providers copied to clipboard

Published 20 hours ago • verified publisher icon astronomer

Fix GKEStartPodOperator in astro-cloud

Open pankajastro opened this issue 2 years ago • 2 comments

By default in astro-cloud the kube namespace is set to deployment namespace Task log

2022-06-15, 03:17:05 UTC] {warnings.py:109} WARNING - /usr/local/airflow/airflow_local_settings.py:94: DeprecationWarning: Accessing configuration method 'get' directly from the configuration module is deprecated. Please access the configuration from the 'configuration.conf' object via 'conf.get'
  pod.metadata.namespace = conf.get("kubernetes", "NAMESPACE")

[2022-06-15, 03:17:07 UTC] {pod_manager.py:137} ERROR - Exception when attempting to create Namespaced Pod: {   "apiVersion": "v1",   "kind": "Pod",   "metadata": {     "annotations": {},     "labels": {       "dag_id": "example_google_kubernetes_engine",       "task_id": "start_pod",       "run_id": "manual__2022-06-15T031138.8582510000-35f0466d7",       "kubernetes_pod_operator": "True",       "try_number": "1",       "airflow_version": "2.3.1-astro.1"     },     "name": "astro-k8s-gke-test-pod-a69aa8dfcc444fc0b9c2384e0892485f",     "namespace": "positional-thermal-0540"   },   "spec": {     "affinity": {},     "containers": [       {         "args": [],         "command": [],         "env": [],         "envFrom": [],         "image": "ubuntu",         "name": "base",         "ports": [],         "resources": {},         "volumeMounts": []       }     ],     "hostNetwork": false,     "imagePullSecrets": [],     "initContainers": [],     "nodeSelector": {},     "restartPolicy": "Never",     "securityContext": {},     "serviceAccountName": "positional-thermal-0540-kpo",     "tolerations": [],     "volumes": []   } }
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 132, in run_pod_async
    resp = self._client.create_namespaced_pod(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
    return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7455, in create_namespaced_pod_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 275, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6e664be9-9de3-4bce-8187-d906f5f543e8', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '0464bffb-79da-48fb-b99e-f85bc4d71a37', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f522c195-ea7b-49a5-a08e-7a9a4c438065', 'Date': 'Wed, 15 Jun 2022 03:17:06 GMT', 'Content-Length': '226'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"positional-thermal-0540\" not found","reason":"NotFound","details":{"name":"positional-thermal-0540","kind":"namespaces"},"code":404}

When setting namespace by setting up AIRFLOW__KUBERNETES__NAMESPACE in dockerfile it trying to use the service account
task log

2022-06-15, 04:30:08 UTC] {warnings.py:109} WARNING - /usr/local/airflow/airflow_local_settings.py:94: DeprecationWarning: Accessing configuration method 'get' directly from the configuration module is deprecated. Please access the configuration from the 'configuration.conf' object via 'conf.get'
  pod.metadata.namespace = conf.get("kubernetes", "NAMESPACE")

[2022-06-15, 04:30:10 UTC] {pod_manager.py:137} ERROR - Exception when attempting to create Namespaced Pod: {   "apiVersion": "v1",   "kind": "Pod",   "metadata": {     "annotations": {},     "labels": {       "dag_id": "example_google_kubernetes_engine",       "task_id": "start_pod",       "run_id": "manual__2022-06-15T042554.0866730000-d2d15677f",       "kubernetes_pod_operator": "True",       "try_number": "1",       "airflow_version": "2.3.1-astro.1"     },     "name": "astro-k8s-gke-test-pod-bad1eab198d441db930b56f414a488ce",     "namespace": "default"   },   "spec": {     "affinity": {},     "containers": [       {         "args": [],         "command": [],         "env": [],         "envFrom": [],         "image": "ubuntu",         "name": "base",         "ports": [],         "resources": {},         "volumeMounts": []       }     ],     "hostNetwork": false,     "imagePullSecrets": [],     "initContainers": [],     "nodeSelector": {},     "restartPolicy": "Never",     "securityContext": {},     "serviceAccountName": "positional-thermal-0540-kpo",     "tolerations": [],     "volumes": []   } }
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 132, in run_pod_async
    resp = self._client.create_namespaced_pod(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
    return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7455, in create_namespaced_pod_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 275, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '052242f8-2a95-45cc-b2fb-59972713a61b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'def9c78c-31df-43e0-a754-09626acf85f7', 'X-Kubernetes-Pf-Prioritylevel-Uid': '6786e3a2-397b-4cbb-889b-e28f28b205f4', 'Date': 'Wed, 15 Jun 2022 04:30:10 GMT', 'Content-Length': '410'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"astro-k8s-gke-test-pod-bad1eab198d441db930b56f414a488ce\" is forbidden: error looking up service account default/positional-thermal-0540-kpo: serviceaccount \"positional-thermal-0540-kpo\" not found","reason":"Forbidden","details":{"name":"astro-k8s-gke-test-pod-bad1eab198d441db930b56f414a488ce","kind":"pods"},"code":403}

pankajastro avatar Jun 15 '22 04:06 pankajastro

Look like GKEStartPodOperatorAsync as well as GKEStartPodOperator currently do not work on astro-cloud

pankajastro avatar Jun 27 '22 09:06 pankajastro

Maybe it would be fix and release in next provider release OSS PR https://github.com/apache/airflow/pull/24658

pankajastro avatar Jul 04 '22 07:07 pankajastro