AWS Glue Job Operator Not Able to Read Jinja Format

[nicholascz666666]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

2.10.1

Apache Airflow version

9.1.0

Operating System

N/A

Deployment

Amazon (AWS) MWAA

Deployment details

I tried to configure the Number Of Workers and worker type within the run_job_kwargs parameter with jinja format like the following run_job_kwargs={"NumberOfWorkers": '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}', "WorkerType":"dummy_type"}

What happened

the value with the jinja format was not recognized during run time and it shows the error that the Number Of Workers should have input type int rather than a string '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}'

What you think should happen instead

'{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}' should pick up the real value during run time

How to reproduce

this could be reproduced by creating a dummy function to push some value to the xcom first then using a glue job operator with specifying run_job_kwargs={"NumberOfWorkers": '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}', "WorkerType":"dummy_type"}

Anything else

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jayceslesar]

Does this persist even if you have render_template_as_native_obj=True to the DAG constructor?

[gopidesupavan]

Yep, this field is not templated. You may raise changes for this, think this is similar to https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/amazon/aws/operators/glue.py#L89 create_job_kwargs

[basvi-chunara]

I would like to work on this. Can this issue be assigned to me?

[potiuk]

Coordinate with @nicholascz666666 @basvi-chunara becasue he was assigned before - but I assigned both of you now.

[basvi-chunara]

okay, thank you for letting me know