data-infra
data-infra copied to clipboard
Create DAG to scrape Littlepay Group and Product information nightly
User story / feature request
As a Cal-ITP Customer Success representative, I need to be able to monitor changes to Littlepay Group and Product configuration in order to help diagnose issues with transactions and configuration related to discounts provided by the Cal-ITP Benefits app.
Acceptance Criteria
- [ ] A table of Littlepay Groups exists containing at least the columns
id, label, participant_id
, populated from the Littlepay Back Office Groups endpoint - [ ] A table of Littlepay Products exists containing at least the columns
id, code, status, type, description, participant_id
, populated from the Littlepay Back Office Products endpoint - [ ] A table of Littlepay Group<>Product linkages exists containing at least the columns
group_id, product_id, participant_id
, populated from the Littlepay Back Office Product Groups endpoint - [ ] The data is acquired nightly and timestamped to allow viewing changes over time
- [ ] This data is available to Metabase users
Notes
The cal-itp/littlepay
Python package implements the necessary APIs to fetch this data as Python objects and in CSV format.
A GitHub Action is set up to scrape this data into CSVs stored in the repository and can be used as a model for a new DAG.
See this Slack thread for more background / context.