Kubernetes decorator

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
       print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

        # Image, name and namespace are all required.
        if not 'image' in kwargs:
            kwargs['image'] = 'python:3.8-slim-buster'

        if not 'name' in kwargs:
            kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'

        if not 'namespace' in kwargs:
            kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

airflow/example_dags/example_kubernetes_decorator.py:32:6: F811 redefinition of unused 'dag' from line 25
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:81:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:84:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:87:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:90:9: F841 local variable 'kwargs_to_upstream' is assigned to but never used

[subkanthi]

airflow/example_dags/example_kubernetes_decorator.py:32:6: F811 redefinition of unused 'dag' from line 25
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:81:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:84:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:87:12: E713 test for membership should be 'not in'
airflow/providers/cncf/kubernetes/decorators/kubernetes.py:90:9: F841 local variable 'kwargs_to_upstream' is assigned to but never used

Fixed these errors.

[subkanthi]

Cant reproduce the mypy errors locally, thoughts @uranusjr

The CI image for Python 3.7 might be outdated

Please run this command at earliest convenience: 

./breeze build-image --python 3.7



Passed```

[potiuk]

Did you dollow the advice @subkanthi ?

./breeze build-image --python 3.7

[subkanthi]

Did you dollow the advice @subkanthi ?

./breeze build-image --python 3.7

Not really, sorry will try again.

[subkanthi]

Did you dollow the advice @subkanthi ?

./breeze build-image --python 3.7

Not really, sorry will try again.

unfortunately that didnt help, will try to troubleshoot the problem.

root@c7681a0b5172:/opt/airflow# mypy airflow/example_dags/example_kubernetes_decorator.py 
Success: no issues found in 1 source file
root@c7681a0b5172:/opt/airflow# mypy --namespace-packages airflow/example_dags
Success: no issues found in 39 source files

[potiuk]

Static checks need fixing

[potiuk]

@uranusjr - looks ready for re-review :)

[subkanthi]

Thanks @josh-fell , updated.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
    jinja_context: dict,
    filename: str,
    render_template_as_native_obj: bool = False,
    template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[potiuk]

Closing, as this has been already implemented and merged in #25663