verified-sources
verified-sources copied to clipboard
support Airflow scheduler, start and end date like Shopify or Zendesk
Which source is this for asana_dlt, facebook_ads,
Describe the data you'd like, be specific, link the docs
- we enable external schedulers for all Incrementals that work with date ranges. dates as strings, datetimes, floats and ints are eligible
- we standardize how the start_date and end_date are passed see below
- we use
TAnyDateTime
to pass date variable and thenensure_pendulum_datetime
to get pendulum date time in UTC - make sure that all Incremental variables are correctly typed with generics
Additional context Airflow scheduler instructions: https://dlthub.com/docs/general-usage/incremental-loading#using-airflow-schedule-for-backfill-and-incremental-loading
See shopify and zenpy for examples on how to pass start and end dates. Most of the resources pass just start date, let's add end date as well (hopefully APIs will support it)