spiceai icon indicating copy to clipboard operation
spiceai copied to clipboard

Add a PartitionedDuckDB Accelerator

Open kczimm opened this issue 9 months ago • 2 comments

📝 Summary

  • Introduce a PartitionedDuckDBAccelerator that is used when DuckDB file mode and partition_by expressions are specified.
  • Implemented partitioning in PartitionTableProvider with insert_into for data partitioning.
  • Created DuckDBPartitionCreator for creating and inferring DuckDB partitions.
  • Refactored create_table_provider in duckdb for reusable table provider creation.
  • Introduced EngineVariant to support partitioned and non-partitioned accelerator engines.
  • Updated AcceleratorEngineRegistry to use EngineVariant for engine registration.

🔗 Related

Part of:

  • #5872 Closes:
  • #6185

🚨 Breaking Changes

📚 Docs

👀 Notes for Reviewers

This does not support reading from PartitionedTableProvider. That will be implemented in #6186

kczimm avatar Jun 29 '25 20:06 kczimm

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: kind/refactor, kind/bug, kind/enhancement, kind/documentation, kind/optimization, kind/dependencies, kind/endgame
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: kczimm

github-actions[bot] avatar Jun 29 '25 20:06 github-actions[bot]

Changing to draft because I found an issue with certain expressions.

kczimm avatar Jun 30 '25 17:06 kczimm

We should have some integration/E2E tests for this - but if you want to track that in another issue, then I'm fine to merge this after fixing the supports_filter_pushdown method to call expr_to_sql

#6386

kczimm avatar Jul 10 '25 14:07 kczimm