v0.0.10 Update Caused Pipelines to Fail

[kosch34]

Hey Ravi,

It looks like the new version that was just pushed immediately failed our DLT META pipeline runs. Seems that it was due to the feautre specifically related to the multi level namespace change Issue

We have since pushed our version back but just letting you know that anyone running their configurations off the previous versions may be failing with this new update. Error looks like this on our end.

com.databricks.pipelines.common.errors.DLTAnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'table_name'.

[ravi-databricks]

@kosch34 , we have added process in FAQ: https://databrickslabs.github.io/dlt-meta/faq/execution/index.html

This failure happens because the pipeline was created using Legacy Publishing mode, which does not support saving tables with catalog or schema qualifiers (such as catalog.schema.table). As a result, using qualified table names leads to an error:

com.databricks.pipelines.common.errors.DLTAnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'catalog_name.schema_name.table_name'

To resolve this, migrate the pipeline to the default (Databricks Publishing Mode) by following Databricks’ guide: Migrate to the default publishing mode.

Alternatively you can tag previous version as below in onboarding job and Pipelines generic notebook: instead of using%pip install dlt-meta use: %pip install dlt-meta==0.0.9