materialize
materialize copied to clipboard
alter_table: Support adding columns to tables
This PR adds support for ALTER TABLE ... ADD COLUMN ...
behind two dyncfgs:
-
enable_alter_table_add_column
-
persist_dangerous_enable_schema_evolution
The Persist dyncfg can be removed once compaction can handle blobs at two different schemas.
The general approach implemented here is to version tables, which builds on top of https://github.com/MaterializeInc/materialize/pull/29647 and https://github.com/MaterializeInc/materialize/pull/29231. The column definitions in a table will have versions attached to them, as well as references to objects, e.g.
CREATE TABLE t1 (a int, b text VERSION ADDED 1);
-- Will only contain column 'a' from 't1'
CREATE VIEW v1 AS SELECT * FROM [u1 as "materialize"."public"."t1" VERSION 0];
The PR is split into separate commits which can be reviewed independently:
- Updates to support versioning of
RelationDesc
s- Changes the in-memory
Table
object to use aVersionedRelationDesc
. This is a wrapper around aRelationDesc
which provides a few methods for adding columns which increments the existing version, and getting theRelationDesc
for a specific version of a table. - Updates Catalog APIs to require specifying a version when getting a
RelationDesc
for an object. To make this change in the least invasive way I introduced a newtrait VersionedCatalogEntry: CatalogEntry
and moved thefn desc(...) -> RelationDesc
method to it. This makes it such that only callers that need aRelationDesc
have to concern themselves with a version, as opposed to all callers of catalog methods. - Plumbing versions to all necessary spots.
- Changes the in-memory
- Updated planning and sequencing to support adding columns to tables.
- This is pretty straight forward, it adds a new
Op::AlterAddColumn
which updates thecreate_sql
for a table and adds a new column withVERSION ADDED <num>
to the table definition.
- This is pretty straight forward, it adds a new
- Makes sure when the
enable_alter_table_add_column
feature is not enabled we don't writeVERSION
s for any objects.- Because we also need to add version references in objects that refer to tables (e.g.
VIEW
s), this commit makes sure when the feature is off we don't try to version things. So as-is this PR should result in 0 changes when the feature is off.
- Because we also need to add version references in objects that refer to tables (e.g.
- Adds some validation to
VERSION
s when specified in a column definition or item reference. -
bin/fmt
Also part of all of these commits are updates to alter-table.slt
to exercise the new functionality.
Motivation
Fixes: https://github.com/MaterializeInc/database-issues/issues/8233
Tips for reviewer
It might be easiest to review this PR commit-by-commit
Checklist
- [ ] This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
- [ ] This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
- [ ] If this PR evolves an existing
$T ⇔ Proto$T
mapping (possibly in a backwards-incompatible way), then it is tagged with aT-proto
label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
- [ ] If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.