Add example DAG for databricks delta support

[dimberman]

Description

What is the current behavior?

The Astro SDK currently does not have an example for loading, transforming, and processing data using Databricks' delta tables.

What is the new behavior?

This PR adds an example to the Astro SDK for loading, transforming, and processing data using Databricks' delta tables. The example includes the following steps:

• Loading data from a CSV file in an s3 bucket into a delta table using the load_file function.
• Transforming the data using Databricks delta SQL SDK and astro's transforn function
• Moving resulting data into a pandas dataframe using the aql.dataframe function.

Does this introduce a breaking change?

No, this is a new feature and does not affect existing functionality.

Checklist

• [ ] Created tests which fail without the change (if possible)
• [ ] Extended the README / documentation, if necessary

[codecov[bot]]

Codecov Report

Patch coverage has no change and project coverage change: +5.98 :tada:

Comparison is base (6ed6ed9) 91.69% compared to head (b1623a3) 97.67%.

:exclamation: Current head b1623a3 differs from pull request most recent head cee0b28. Consider uploading reports for the commit cee0b28 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1585      +/-   ##
==========================================
+ Coverage   91.69%   97.67%   +5.98%     
==========================================
  Files          72       23      -49     
  Lines        4250      904    -3346     
  Branches      511        0     -511     
==========================================
- Hits         3897      883    -3014     
+ Misses        257       21     -236     
+ Partials       96        0      -96     

Flag	Coverage Δ
PythonSDK	?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 95 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[kaxil]

@dimberman Can you get this to the finish line please?

[dimberman]

@tatiana @kaxil so this PR runs fine in my local set up but it fails in CI due to https://github.com/astronomer/astro-sdk/issues/1744. Should we wait until we've solved #1744 before merging this PR or merge now and then solve?

[tatiana]

@dimberman, it would be great if we could address CI issue #1744. before merging this..! How is that going?

[phanikumv]

waiting on https://github.com/astronomer/astro-sdk/issues/1744