materialize icon indicating copy to clipboard operation
materialize copied to clipboard

sql-parser: move minor `CREATE SOURCE` options to `WITH` block

Open andrioni opened this issue 2 years ago • 2 comments

More context in Slack, but the main idea is to move options like REMOTE and TIMELINE to the new WITH (OPT1 VAL1 OPT2 VAL2) block that supports order-independent settings, similar to CREATE CONNECTION already works.

andrioni avatar Jul 27 '22 18:07 andrioni

#14082 is done, pulling REMOTE to live with SIZE in the new WITH block.

I'm trying to figure out which other options we'd want to roll in here, if any. timeline lives in the legacy WITH block; is the idea that we'd roll everything that lives in legacy_with_options over to the new format and style? Only hoist timeline over? Some other thing?

cc @benesch as the original idea haver.

bkirwi avatar Aug 08 '22 15:08 bkirwi

We should cleanly separate the options! Some of the legacy with options (like start_offset) ought to move to a Kafka-specific option block. Only the options that are unrelated to the connector, format, or envelope should move to the new top-level WITH block. I think that's just REMOTE, SIZE, and TIMELINE, but not sure. TIMELINE is also not something we want to expose to users quite yet, I don't think, so no particular urgency to move it.

benesch avatar Aug 10 '22 05:08 benesch

REMOTE, SIZE, and TIMELINE have been moved, and we've decided against moving epoch_ms_timeline... so it looks like we're done here!

There are a few other odds and ends that are moving to source- or envelope-specific option blocks; see https://github.com/MaterializeInc/materialize/issues/14341 for some related work.

bkirwi avatar Aug 25 '22 22:08 bkirwi