George Zubrienko

We add jars under native user and everything works fine. ``` USER 1001 # Azure SDKs, Hadoop-Azure, Delta, Wildfly SSL, common jars (delta + azure hadoop) RUN wget --quiet https://repo1.maven.org/maven2/com/microsoft/azure/adal4j/1.6.7/adal4j-1.6.7.jar...

We use thrift server and create tables the same way, with jars backed inside spark image and provided to spark as @zsxwing suggested. No issues with delta 1.1.

Finally found the same issues as ours! Tried 2.4.0, doesn't help, tasks still stuck in queued with the same behaviour. Downgrade to 2.3.3 fixes the issue. Also with cncf-kubernetes 4.3.0,...

I will try it this week around Wednesday. Was on vacation :)

got a bit delayed, will test tomorrow and come back with results. Sorry.

@potiuk We no longer observe the issue on the following setup: Kubernetes Executor (1.22.11) / Airflow 2.4.1-python3.9 / Helm Chart 1.7.0 with dag processor enabled.

@stevenpackardblp please update your PR and we can merge ;)

@stevenpackardblp another ping :) if no reply in a week I'll take over this PR and re-create it

@dax hello again :) could please sync your branch, so we can merge the PR?

I think the library behaviour is correct in this case, otherwise you break ser-deser consistency. I think in your case, just remapping values to another class should do the trick....