meltano
feat: configurable target schema change settings by default
The target sdk sql connector class has allow_column_alter which is set by the target developer in the connector implementation. We should make this a configuration option thats available to the user by default. It will let us have a consistent enum of options available to the user with a consistent config name instead of letting each developer implement it themselves in slightly different ways.
Related
- https://github.com/meltano/sdk/issues/1783
- https://github.com/meltano/sdk/pull/2504
- https://github.com/meltano/sdk/pull/2505
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Still relevant. @edgarrmondragon this might be something we want to have available as an option in Arch so having a way to control it externally would be nice eventually.
Still relevant. @edgarrmondragon this might be something we want to have available as an option in Arch so having a way to control it externally would be nice eventually.
Makes sense. Should be pretty quick to implement too fwiw.
Would love to be able to modify my schemas outside of meltano and just have meltano "bootstrap" schemas, and then adjust accordingly as I manage things on my end.
Load it all in as text initially for example, but if I come and change it to int later, just roll with it and follow the new rule(s).
Would love to be able to modify my schemas outside of meltano and just have meltano "bootstrap" schemas, and then adjust accordingly as I manage things on my end.
Load it all in as
textinitially for example, but if I come and change it tointlater, just roll with it and follow the new rule(s).
Thanks for your input @atrauzzi! That makes me think we would need 3 schema update modes:
- adapt: the incoming type takes priority and the existing column type is updated
- preserve (your suggestion): the existing type takes priority and the incoming type is ignored
- fail: if there's a mismatch between the incoming and existing types, we fail
Wdyt @pnadolny13?
Yeah that sounds good. The overall goal in my scenarios are to not have to configure my schema explicitly in meltano. Instead, I want to delegate that entirely to meltano. That way if my source data schema changes, I'm not having to make config changes.
I might rename your two options to be a bit more explicit, as the effect can be ambiguous depending on perspective:
- sync-target-schema
- prefer-target-schema
Also, two friendly reminders:
- With prefer-target-schema, I would still rely on meltano to create the newly discovered columns.
- I might make columns nullable when I'm manually adjusting types. Might be nice if meltano could support that, versus empty strings