spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

[FEA] Support metadata row index for Delta tables with DVs

Open jihoonson opened this issue 4 weeks ago • 3 comments

Is your feature request related to a problem? Please describe.

In Delta tables with DVs, spark.databricks.delta.deletionVectors.useMetadataRowIndex is the configuration to enable optimizations such as file split and predicate pushdown. Since the plugin doesn't support this configuration yet, those optimizations are disabled for Delta tables with DVs. This can cause various issues including performance problems. Especially the missing file split support can lead to data skew, which in turn can lead to not only slow queries but also out of memory errors or cudf column size overflow errors.

Describe the solution you'd like

The plugin should support spark.databricks.delta.deletionVectors.useMetadataRowIndex.

Describe alternatives you've considered

N/A

Additional context

N/A

jihoonson avatar Dec 12 '25 02:12 jihoonson