databend
databend copied to clipboard
feat: basic support of schema evolution in copy for parquet
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
feats:
- table alter and data append is committed together atomicly.
example:
# prepare data
create or replace table t1(a int);
create or replace stage s1;
copy into @s1 from (select 1 as a, 2 as b, 3 as c)
# ENABLE_SCHEMA_EVOLUTION
ALTER TABLE t1 SET options(ENABLE_SCHEMA_EVOLUTION = true);
# do copy
copy into t1 from @s1/ file_format=(type=parquet);
not supported yet:
- adding field in tuple type
- merge types, e.g. int8 and int16
Tests
- [ ] Unit Test
- [x] Logic Test
- [ ] Benchmark Test
- [ ] No Test - Explain why
Type of change
- [] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):