(draft) poc for better UC integration

[desmondcheongzx]

Changes Made

Was curious as to why our APIs weren't working too hot with UC and took a look. It seems that the daft.unity_catalog.UnityCatalog object we pass into from_unity is more like a UC client rather than a UC catalog itself.

One possibility to address this is to change our API to from_unity(client, catalog_name).

With this change we can finally do this:

import os
import daft
from daft import Session
from daft.unity_catalog import UnityCatalogClient

unity_client = UnityCatalogClient(
    endpoint=os.environ["EVENTUAL_DATABRICKS_ENDPOINT"],
    token=os.environ["DATABRICKS_API_KEY"]
)

sess = Session()
catalog = daft.Catalog.from_unity(unity_client, "jaytest-unity")
sess.attach(catalog)
print(sess.current_catalog())
sess.sql('USE "jaytest-unity".default')
sess.sql("select * from nation").show()

Related Issues

Related to #4059

Checklist

• [ ] Documented in API Docs (if applicable)
• [ ] Documented in User Guide (if applicable)
• [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
• [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 59.09091% with 9 lines in your changes missing coverage. Please review.

Project coverage is 78.07%. Comparing base (5fd1f2f) to head (c761fff). Report is 263 commits behind head on main.

Files with missing lines	Patch %	Lines
daft/catalog/__unity.py	47.05%	9 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4145      +/-   ##
==========================================
+ Coverage   76.50%   78.07%   +1.56%     
==========================================
  Files         790      793       +3     
  Lines      102622   102281     -341     
==========================================
+ Hits        78514    79854    +1340     
+ Misses      24108    22427    -1681     

Files with missing lines	Coverage Δ
daft/catalog/__init__.py	73.01% <100.00%> (ø)
daft/unity_catalog/__init__.py	100.00% <100.00%> (ø)
daft/unity_catalog/unity_catalog.py	25.00% <100.00%> (+0.86%)	:arrow_up:
daft/catalog/__unity.py	50.00% <47.05%> (ø)

... and 49 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.