verified-sources icon indicating copy to clipboard operation
verified-sources copied to clipboard

support Airflow scheduler, start and end date like Shopify or Zendesk

Open rudolfix opened this issue 1 year ago • 0 comments

Which source is this for asana_dlt, facebook_ads,

Describe the data you'd like, be specific, link the docs

  1. we enable external schedulers for all Incrementals that work with date ranges. dates as strings, datetimes, floats and ints are eligible
  2. we standardize how the start_date and end_date are passed see below
  3. we use TAnyDateTime to pass date variable and then ensure_pendulum_datetime to get pendulum date time in UTC
  4. make sure that all Incremental variables are correctly typed with generics

Additional context Airflow scheduler instructions: https://dlthub.com/docs/general-usage/incremental-loading#using-airflow-schedule-for-backfill-and-incremental-loading

See shopify and zenpy for examples on how to pass start and end dates. Most of the resources pass just start date, let's add end date as well (hopefully APIs will support it)

rudolfix avatar Aug 06 '23 18:08 rudolfix