data-infra icon indicating copy to clipboard operation
data-infra copied to clipboard

Handle JSON & new columns in Airtable `fare_systems` in dbt

Open lauriemerrell opened this issue 1 year ago • 0 comments

As far as I know, no one is using fare systems data in the warehouse at time of writing (November 2023). This ticket will only become relevant if that changes.

As a data user who needs to access Airtable fare systems data in the data warehouse, I need the schemas for that data to be updated in dbt so that I can use new and changed fields in the warehouse

PR #3067 changed some columns to JSON type and added new columns to the external table for fare systems that haven't been added to dbt yet. If we want these columns to be available and usable in the data warehouse, we will need to make dbt-side updates to handle these columns (unpack/cast JSON and add the new columns).

Here's a summary of the changes in the external table, which will need to be handled on the dbt side if anyone wants access to this data in the warehouse:

column_name change_type change_details
distance_fares new_column
fare_account new_column
fare_capping type_change STRING to JSON
fare_media new_column
fare_product_price_max new_column
fare_product_price_min new_column
fare_products new_column
fare_products_notes new_column
free_fare new_column
interagency_xfer_discounts new_column
interagency_xfer_notes new_column
interagency_xfer_systems new_column
intra_system_xfer_cost new_column
intra_system_xfer_discounts new_column
intra_system_xfer_time new_column
market_based_fares new_column
num_reduced_fare_categories new_column
payment_accepted type_change STRING to JSON
reduced_fare_categories type_change STRING to JSON
route_fares new_column
smartcard_discount new_column
ticket_media type_change STRING to JSON
ticket_pass_sales_methods type_change STRING to JSON
ticket_validation type_change STRING to JSON
youth_specific_fare new_column
zone_fares new_column

lauriemerrell avatar Nov 02 '23 19:11 lauriemerrell