Subham Singhal

How do we pass tenantId info when we are switching thread

1

TenantId is stored in a thread local variable. Now if we switch thread either by using ExecutorService or Vertx event then how can we pass tenant info to other thread?

Enable idempotent write in BQ. This is similar to Databricks idempotent write. [link](https://docs.delta.io/latest/delta-streaming.html#idempotent-table-writes-in-foreachbatch)

### Description Currently if we need to pass any parameter to spark job we need to hardcode parameters in json config. But we want to pass parameters dynamically through airflow....

Feature/pass airflow confi as job param

1

Fixes: #39002 ### Tests: 1. empty airflow config and empty job parameter -- empty job parameter 2. empty airflow config and some job parameter in databricks operator -- job params...

[SPARK-46798][Structured Streaming] Kafka custom partition location assignment (rack awareness)

21

This is rework of https://github.com/apache/spark/pull/44954 What changes were proposed in this pull request? Add support for custom partition location assignment for Kafka sources in Structured Streaming. Why are the changes...