logica Databricks SQL support

Based on recent work for clients, I for one would think support for Databricks SQL would be a huge deal. Seems to be used a little bit of everywhere.

I recently learned Databricks has teamed up with dbt to enable using their SQL templating tech for simplifying writing transformation logic, but Logica would be on a completely different level in my opinion.

Feb 15 '23 14:02 samuell

Thanks for filing the issue!

I didn't work with Databricks before. I briefly looked at documentation.

The good news is that Databricks has structs and arrays, so adding this as a Logica engine appears doable.

Unfortunately it looks like Databricks doesn't have a free tier. The free trial is limited to 14 days. So to implement it we'd need somebody who already pays for Databricks.

Drop a note in this issue if you are interested in contributing to this.

Feb 21 '23 17:02 EvgSkv

Hi, @samuell and @EvgSkv

My team and I also looked into DatabricksSQL support, and we're doing some work right now to target Databricks at least partially. Some of it might be a bit crude, but we'd be happy to raise a PR to see if it could be useful to others once we're done.

I'll follow up on this within several weeks.

Jul 22 '23 17:07 rvalek

Hi @rvalek , that's great to hear! Adding nuances of Databricks to dialects (under compile folder) should be straightforward and safe. If deeper changes are needed (e.g. in universe.py or expr_translate.py), I'm happy to work with you to get them done. Looking forward!

Jul 26 '23 20:07 EvgSkv

Hi! We are a little behind on this promise due to some unrelated technical issues, but we are close to resolving them. Just wanted update here that we are still interested in sharing our work on the Databricks dialect. We're planning to raise the PR in early-mid September.

Aug 22 '23 14:08 rvalek

hi @EvgSkv! can you suggest in which branch it is best to open a PR for this?

Aug 31 '23 15:08 KantorSerhiy

Hi @KantorSerhiy ! Sorry, missed the comment. Sending PR to main is just fine! Thank you!

Sep 06 '23 15:09 EvgSkv

@KantorSerhiy , btw, I did respond in the PR, it looks great! I left just a couple of requests to comment and it's good to be merged after that.

Sep 12 '23 16:09 EvgSkv

hey @EvgSkv! yes, I saw your comments and will definitely deal with them. but it takes for me 1-2 weeks=( Thank you!

Sep 12 '23 16:09 KantorSerhiy

@KantorSerhiy great! No worries! Just wanted to make sure your progress isn't blocked on me :-)

Sep 12 '23 23:09 EvgSkv

Hi @samuell , we merged the baseline databricks dialect a couple of weeks ago. Feel free to try it out!

Oct 19 '23 14:10 rvalek