spiceai
spiceai copied to clipboard
Add a PartitionedDuckDB Accelerator
📝 Summary
- Introduce a
PartitionedDuckDBAcceleratorthat is used when DuckDB file mode and partition_by expressions are specified. - Implemented partitioning in
PartitionTableProviderwithinsert_intofor data partitioning. - Created
DuckDBPartitionCreatorfor creating and inferring DuckDB partitions. - Refactored
create_table_providerin duckdb for reusable table provider creation. - Introduced
EngineVariantto support partitioned and non-partitioned accelerator engines. - Updated
AcceleratorEngineRegistryto useEngineVariantfor engine registration.
🔗 Related
Part of:
- #5872 Closes:
- #6185
🚨 Breaking Changes
📚 Docs
👀 Notes for Reviewers
This does not support reading from PartitionedTableProvider. That will be implemented in #6186
✅ Pull with Spice Passed
Passing checks:
- ✅ Title meets minimum length requirement (10 characters)
- ✅ Has at least one of the required labels:
kind/refactor,kind/bug,kind/enhancement,kind/documentation,kind/optimization,kind/dependencies,kind/endgame - ✅ No banned labels detected
- ✅ Has at least one assignee:
kczimm
Changing to draft because I found an issue with certain expressions.
We should have some integration/E2E tests for this - but if you want to track that in another issue, then I'm fine to merge this after fixing the
supports_filter_pushdownmethod to call expr_to_sql
#6386