databricks
1.11.3 New warning CustomKeyInObjectDeprecation for "Custom key `databricks_tags`"
As we all know, it is possible to use Databricks Column Tags in both schema.yml and as config. The documentation gives this example for apply the tag data_classification with the value public:
models:
- name: customers
columns:
- name: customer_id
databricks_tags:
data_classification: "public"
However in version 1.11.3 and at least in 1.11.2, a new warning is raised
Custom key `databricks_tags` found at `models[1].columns[1]` in file
`models/customers/schema.yml`. This may mean the key is a typo, or
is simply not a key supported by the object.
It still works, but the warning seems to indicate that it won't work for much longer. What is the proper way to run going forward?
Debug information
13:49:44 Running with dbt=1.11.0-rc3
WARNING:thrift.transport.sslcompat:using legacy validation callback
13:49:47 Registered adapter: databricks=1.11.3
When running the task with version 1.10.12, we instead get this debug information
05:05:56 Running with dbt=1.10.11
WARNING:thrift.transport.sslcompat:using legacy validation callback
/databricks/python/lib/python3.12/site-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
warnings.warn(message, UserWarning)
05:05:58 Registered adapter: databricks=1.10.12
There seems to be some documentation about this on the official site.
They seem to indicate that it can be resolved using
columns:
- name: my_column
config:
meta:
some_key: some_value
where some_key would be databricks_tags. Has anyone tried?
I'm pretty sure that will not work. @dataders, how should this be fixed in the adapter? Is there somewhere we can declare it as "in spec", because otherwise this is going to be pure hell on adapter maintainers.
