Ability to import dlt signatures without a current SparkSession

[marijncv]

I've been happily using the dlt-with-debug library, but I'm running into an issue when importing the dlt signatures without an active SparkSession. I'm trying:

import dlt_with_debug.dlt_signatures as dlt

But the __init__.py calls the v2.py file which in turn tries to get the pipeline_id from the current SparkSession. But this fails when there is no active SparkSession.

I think my usecase can be achieved with an extra check in https://github.com/souvik-databricks/dlt-with-debug/blob/db7f9b3666d000d96908eb5cac60dddf86ac8c18/dlt_with_debug/v2.py#L8 that first confirms there is an actual SparkSession before calling .conf.get() on it.

@souvik-databricks curious what you think! And I'd be happy to contribute via PR

[souvik-databricks]

Hi @marijncv Thank you for using the project and I saw your PR for the fix. Will review it and get it in by the end of this week.

[javamonkey79]

I am having the same issue, could you please merge the PR?

[javamonkey79]

@marijncv since @souvik-databricks is MIA, could you bumpversion in your fork?