astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Published 20 hours ago • verified publisher icon astronomer

Map File & Table classes to Datasets (AIP-48)

Open kaxil opened this issue 1 year ago • 1 comments

There will be an alpha build of 2.4 very soon (Sync with the Ash/Jed to get the tag), so to make sure Python SDK can use 2.4 dataset feature (and allows viewing dataset dependency in UI), let's map the File and Table to AIP-48 datasets to show dependencies between tasks and Dags.

kaxil avatar Aug 08 '22 00:08 kaxil

Astro Runtime 6.0.0-alpha1 image is out where I will do the mapping

kaxil avatar Aug 12 '22 10:08 kaxil

WIP diff at https://github.com/astronomer/astro-sdk/compare/main...aip-48-mapping Branch at https://github.com/astronomer/astro-sdk/tree/aip-48-mapping

image
image

kaxil avatar Aug 22 '22 21:08 kaxil

Both Table and File object have been mapped with 6.0 alpha2 runtime image but the code needs to be cleaned up

kaxil avatar Aug 25 '22 03:08 kaxil

PR: https://github.com/astronomer/astro-sdk/pull/786

Pending items:

  • [x] Add inlets and outlets on all operators
  • [ ] Add docs

kaxil avatar Sep 06 '22 21:09 kaxil

PR is in review and @utkarsharma2 is going to review it. Potentially merge it if looks all good today.

pankajkoti avatar Sep 07 '22 06:09 pankajkoti

Another PR: https://github.com/astronomer/astro-sdk/pull/808

Only doc PR is pending now

kaxil avatar Sep 09 '22 01:09 kaxil

I will create separate task for docs so that I can close this out

kaxil avatar Sep 12 '22 11:09 kaxil