astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Play nicely with ExternalPythonOperator

Open collinmcnulty opened this issue 3 years ago • 0 comments

Please describe the feature you'd like to see Will Astro SDK be likely to play nicely out of the box with ExternalPythonOperator? Having to choose between environment isolation and the improved dag authoring has been a thorn in the side for a while, and if these two work well together that would be a big win.

I think it might already play nicely as long as what the author returns from the SDK tasks is a dataframe and not a Table.

Describe the solution you'd like I want to do some transformations in python with a library that is incompatible with runtime (e.g. dbt-core) on a table in snowflake that I want to load with astro sdk.

Are there any alternatives to this feature? Maybe a Table-to-dataframe method?

Additional context It is unclear to me if SDK's XCOMArg objects will "just work" as an input to an ExternalPythonOperator task, or if it needs to be an SDK-provided operator that implements the external python logic. The former would be better.

collinmcnulty avatar Sep 06 '22 18:09 collinmcnulty