data-infra
data-infra copied to clipboard
Handle JSON & new columns in Airtable `fare_systems` in dbt
As far as I know, no one is using fare systems data in the warehouse at time of writing (November 2023). This ticket will only become relevant if that changes.
As a data user who needs to access Airtable fare systems data in the data warehouse, I need the schemas for that data to be updated in dbt so that I can use new and changed fields in the warehouse
PR #3067 changed some columns to JSON
type and added new columns to the external table for fare systems that haven't been added to dbt yet. If we want these columns to be available and usable in the data warehouse, we will need to make dbt-side updates to handle these columns (unpack/cast JSON and add the new columns).
Here's a summary of the changes in the external table, which will need to be handled on the dbt side if anyone wants access to this data in the warehouse:
column_name | change_type | change_details |
---|---|---|
distance_fares | new_column | |
fare_account | new_column | |
fare_capping | type_change | STRING to JSON |
fare_media | new_column | |
fare_product_price_max | new_column | |
fare_product_price_min | new_column | |
fare_products | new_column | |
fare_products_notes | new_column | |
free_fare | new_column | |
interagency_xfer_discounts | new_column | |
interagency_xfer_notes | new_column | |
interagency_xfer_systems | new_column | |
intra_system_xfer_cost | new_column | |
intra_system_xfer_discounts | new_column | |
intra_system_xfer_time | new_column | |
market_based_fares | new_column | |
num_reduced_fare_categories | new_column | |
payment_accepted | type_change | STRING to JSON |
reduced_fare_categories | type_change | STRING to JSON |
route_fares | new_column | |
smartcard_discount | new_column | |
ticket_media | type_change | STRING to JSON |
ticket_pass_sales_methods | type_change | STRING to JSON |
ticket_validation | type_change | STRING to JSON |
youth_specific_fare | new_column | |
zone_fares | new_column |