DataSQRL
Support a root configuration file
Some users have, on the order of, 20ish source tables that are defined in their source package. They have multiple copies of packages for kafka and local. That means they have about 50 .table.json configuration files. Most of the content in those files is identical: the flink connector configuration, the watermark, the metadata for kafka. Even the timestamp is mostly the same.
We can make those configuration files DRYer by having a root package-name.shared.json file where all the shared configuration is stored. On import, the table.json file gets merged with the .shared.json (with the former taking precedence) before reading it. That’s exactly the same thing we do with package.json files and the infrastructure exists already.
Can we punt on this in favor of supporting flink create table statements?
For 0.6 we are moving to CREATE TABLE statements so we support all of FlinkSQL including catalog definitions, LIKE and so forth. But we still want to think about how we structure this in the filesystem. It's nice to have one table definition per file since that allows us to import individual tables and rename them on import. In this world, we could consider a "shared" sql file that is always loaded (but only once) for imports from a particular package.
We are going to punt this in favor of importing into separate databases.