astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Add examples of how to use operators/decorators in docstrings

Open tatiana opened this issue 3 years ago • 2 comments

Context

In the past, we had a tutorial which illustrated how to use each of our operators/decorators: https://github.com/astronomer/astro-sdk/blob/be6280df00ccff0d7a1c0dfb099b2065303dbe88/REFERENCE.md

Since we started auto-generating the documentation with Sphinx, the overall description of each operator is incomplete: Screen Shot 2022-07-27 at 3 16 41 PM

Depends on:

  • [x] #586
  • [x] #587
  • [x] #588
  • [ ] #589
  • [x] #590
  • [x] #591
  • [x] #592
  • [ ] #593
  • [x] #594

Acceptance criteria

Have a reference page per operator/decorator similar to https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/ecs.html#howto-operator-ecsoperator

In which we reference parts of an (automated tested) example DAG which illustrates the usage of that operator/decorator.

Many of these use cases already exist in our example DAGs - we should reference them.

  • [X] load_file, including examples - story points: 5
    • (HTTP + CSV) loading to a table
    • load to dataframe
    • native load with BQ
    • native load with Snowflake
    • native load with fallback
    • non-native support
    • specifying the schema (tables.columns)
  • [x] transform - story points: 2
    • arguments are tables, illustrating templating
    • one argument is a dataframe
  • [X] drop_table - story points: 1
    • single example
  • [ ] run_raw_sql - story points: 2
    • without handler function
    • with handler function
  • [X] append - story points: 1
    • single example
  • [ ] merge - story points: 2
    • with conflicts (what options the user has)
    • without conflicts
  • [x] export_file - story points: 2
    • exporting from a table
    • exporting from a dataframe
  • [ ] dataframe - story points: 1
    • single example
  • [X] cleanup - story points: 1
    • single example

If no example DAGs or operators are illustrating the functionalities, we should create them.

tatiana avatar Jul 28 '22 08:07 tatiana

@tatiana can we add story points to it?

utkarsharma2 avatar Aug 11 '22 15:08 utkarsharma2

@utkarsharma2 This one is an "epic" that encompasses several docs issues as listed in the PR description so we don't need to add story points to this task

kaxil avatar Aug 12 '22 12:08 kaxil

All of the subtasks have been completed. Closing this issue.

utkarsharma2 avatar Aug 18 '22 17:08 utkarsharma2