databend icon indicating copy to clipboard operation
databend copied to clipboard

Feature: Support PARTITION BY With Unloads

Open MicahLyle opened this issue 3 months ago • 1 comments

Summary

Snowflake allows:

COPY INTO { internalStage | externalStage | externalLocation }
     FROM { [<namespace>.]<table_name> | ( <query> ) }
[ PARTITION BY <expr> ]

From looking at Databend's documentation, it seems like this is not supported. If it is, then I think it's a simple documentation update.

Rationale On Databend's 2025 roadmap is being an on-prem replacement for Snowflake. This is a feature that would bring Databend closer to feature parity with Snowflake.

MicahLyle avatar Sep 17 '25 14:09 MicahLyle

I thought of these changes:

  • src/query/ast/src/ast/statements/copy.rs for CopyIntoLocationStmt grammar definition
  • src/query/sql/src/planner/plans/copy_into_table.rs and src/query/sql/src/planner/plans/copy_into_location.rs support new keywords
  • src/query/service/src/interpreters/interpreter_copy_into_location.rs prepare physical plan
  • src/query/service/src/physical_plans/physical_copy_into_location.rs use PipelineBuilder to write data. Here is the problem: build_append2table_with_commit_pipeline use table API, but does not have any partition write API ...Do I need to add an API for it? Or do you have any good suggestions? @wubx

camilesing avatar Oct 21 '25 15:10 camilesing