materialize icon indicating copy to clipboard operation
materialize copied to clipboard

alter_table: Support adding columns to tables

Open ParkMyCar opened this issue 5 months ago • 4 comments

This PR adds support for ALTER TABLE ... ADD COLUMN ... behind two dyncfgs:

  1. enable_alter_table_add_column
  2. persist_dangerous_enable_schema_evolution

The Persist dyncfg can be removed once compaction can handle blobs at two different schemas.

The general approach implemented here is to version tables, which builds on top of https://github.com/MaterializeInc/materialize/pull/29647 and https://github.com/MaterializeInc/materialize/pull/29231. The column definitions in a table will have versions attached to them, as well as references to objects, e.g.

CREATE TABLE t1 (a int, b text VERSION ADDED 1);

-- Will only contain column 'a' from 't1'
CREATE VIEW v1 AS SELECT * FROM [u1 as "materialize"."public"."t1" VERSION 0];

The PR is split into separate commits which can be reviewed independently:

  1. Updates to support versioning of RelationDescs
    1. Changes the in-memory Table object to use a VersionedRelationDesc. This is a wrapper around a RelationDesc which provides a few methods for adding columns which increments the existing version, and getting the RelationDesc for a specific version of a table.
    2. Updates Catalog APIs to require specifying a version when getting a RelationDesc for an object. To make this change in the least invasive way I introduced a new trait VersionedCatalogEntry: CatalogEntry and moved the fn desc(...) -> RelationDesc method to it. This makes it such that only callers that need a RelationDesc have to concern themselves with a version, as opposed to all callers of catalog methods.
    3. Plumbing versions to all necessary spots.
  2. Updated planning and sequencing to support adding columns to tables.
    1. This is pretty straight forward, it adds a new Op::AlterAddColumn which updates the create_sql for a table and adds a new column with VERSION ADDED <num> to the table definition.
  3. Makes sure when the enable_alter_table_add_column feature is not enabled we don't write VERSIONs for any objects.
    1. Because we also need to add version references in objects that refer to tables (e.g. VIEWs), this commit makes sure when the feature is off we don't try to version things. So as-is this PR should result in 0 changes when the feature is off.
  4. Adds some validation to VERSIONs when specified in a column definition or item reference.
  5. bin/fmt

Also part of all of these commits are updates to alter-table.slt to exercise the new functionality.

Motivation

Fixes: https://github.com/MaterializeInc/database-issues/issues/8233

Tips for reviewer

It might be easiest to review this PR commit-by-commit

Checklist

  • [ ] This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • [ ] This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • [ ] If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • [ ] If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

ParkMyCar avatar Sep 20 '24 21:09 ParkMyCar