Merel Theisen

> Deprecation warnings will be issued in advance for any features or Python versions that are being removed. How do you plan to do this? In a Slack announcement or...

This PR has been open for a long time and diverged from the `main`/`develop` branch. I'll close it for now and suggest that if @bpmeek or anyone else wants to...

It's been a while since I looked at the issue this PR is for, but if I understand correctly the problem with APIDataset is that it only returns the response,...

Hi @star-yar, thanks for flagging this issue. It sounds like the best solution is to allow passing a session to the dataset or you could monkey patch it. We'd be...

Glad you managed to get it working @star-yar ! I was thinking maybe nesting it so something like: ``` dataset: type: spark.SparkDataset spark_session_builder: profile: profile_name serverless: True ```

Hmm yes that's a good point. It's not great to bloat the catalog and it's technically not really dataset level config, but higher level. It makes sense to expand the...

> Any ideas why this might fail? Not sure this is caused by my PR, wdyt? > > @merelcht No this isn't related to your PR. This failure started showing...

@star-yar, the test failures are resolved, but now there's a coverage issue which is related to the changes you made. Can you add tests to cover the behaviour?

Hi @star-yar , I'm really sorry for the late response. You raise a very good point. In fact, this is what we recommend when using spark (without databricks) in the...

Hi @hugodscarvalho are you still interested in finishing this PR?