public-datasets-pipelines
public-datasets-pipelines copied to clipboard
chore(main): release 6.0.0
:robot: I have created a release beep boop
6.0.0 (2024-10-28)
⚠ BREAKING CHANGES
- Remove support for Airflow 1 (#447)
cleanup
Features
- Add
cycle_hire
to London Bicycles dataset (#596) (889c47c) - Add Year 2023 To NOAA GSOD Pipelines (#564) (5764649)
- Adding pipelines to NOAA Dataset (#531) (0989924)
- Create load process to onboard NPPES npi_raw table data (#567) (2a8bbba)
- IDC v12 release (#530) (9ab5ad6)
- Migrate the Xenon dataset Covid19 JHU (#525) (9e93676)
- Migrate the Xenon dataset Covid19 Symptom Search (#534) (09b8f34)
- Migrate the Xenon dataset USDA NASS Agriculture (#547) (f1001f3)
- Onboard Austin 311 Service Requests Dataset (#526) (9c864d3)
- Onboard Census Bureau International Dataset (#425) (2322861)
- onboard chicago taxi trips dataset (#533) (4ee3b84)
- Onboard Clemson Dice Traffic Vision Dataset (#475) (f62d4f5)
- Onboard dataset MIMIC-III (#456) (bd1582a)
- Onboard GHCN_M Pipelines to NOAA dataset (#545) (82c7616)
- Onboard Hacker News dataset (#532) (788273f)
- Onboard Libraries IO dataset-2 (#551) (93063b9)
- Onboard Llibraries IO dataset (#540) (950a290)
- onboard london cycle stations dataset (#537) (f81d1c6)
- onboard NIH GUDID historic data dataset (#544) (2656e49)
- Onboard NLM RXNorm Dataset (#605) (80bd8c0)
- Onboard the dataset Open Buildings v2 (#553) (a18bd17)
- Onboard Uniref50 Protein data (#524) (f1cfae2)
- Onboard US Climate Normals Dataset (#446) (42caa82)
- Onboard Visual Questions & Answers Dataset (#405) (fae3306)
Bug Fixes
- Add bike type field to Austin bikeshare trips schema. (#609) (541751d)
- Add IDC v13 datasets. (#561) (70c0aba)
- Add some fields and remove some fields from google_political_ads dataset. (#661) (8fc022a)
- Adding v4. (#566) (5e64914)
- Change bucket variable for USDA NASS Agriculture (#552) (7b476cc)
- Change DAG frequency for the dataset Covid19 Symptom Search (#546) (89b54eb)
- Change schedule for EUMETSAT pipeline (#585) (9909d6a)
- Change schedule of open targets dataset updates to 00:01 on 1st of each month. (#611) (b2a6756)
- Changed schedule time to every 12 hours for cloud_storage_geo_index. (#599) (fbbd4b4)
- Extended transfer timeout for EUMETSAT dag pipelines. (#590) (a00ddc6)
- fix to omission google political ads. (#674) (fbd5ec2)
- Fixes to dags in new v2-2024 environment (#760) (e8f0240)
- Fixing a small typo (#474) (5b7e8d9)
- Increased chunksize due to table partition quota exceeded errors during ETL execution. (#837) (cfb8732)
- Iowa liquor sales fix 20230828 (#623) (5007547)
- lint errors (#619) (b5f342c)
- Migrate Austin DAGs (#812) (adbba4b)
- Migrate bls 20241007 (#829) (4e4c1fa)
- Migrate CDC Dags to new environment. (#832) (72e5c4b)
- Migrate CMS Inpatient and outpatient dags to new environment. (#830) (3a9c735)
- Migrate Covid 19 DAGs to new environment. (#833) (a5208bf)
- Migrate EBI pipelines to new environment. (#838) (54040f3)
- Migrate FEC Dags to new environment (#843) (12325d6)
- Migrate IMDB to new environment. (#844) (4a64e72)
- Migrate to new environment. (#845) (141872b)
- Migration changes for fda_drug_enforcement DAG (#842) (f51b142)
- Migration changes for London Bicycles DAG. (#836) (f833866)
- Migration for FDIC dags. (#846) (32b1da8)
- Migration of DAG to v2-2024 environment - relevant changes (#786) (a8eb762)
- Migration to new environment. (#840) (337e69c)
- Missing commas in pipeline.yaml in google_political_ads dataset. (#671) (f224a89)
- modified source file location (#538) (f602101)
- Multiple fixes to San Francisco dataset. (#618) (78261b9)
- namespace fix to inpatient_charges. (#831) (3177968)
- Overwrite destination table for open_targets BQ copy operation (#548) (9183019)
- Re-engineer and Migrate Hacker News DAG (#762) (920332f)
- Re-engineer UniRef 50 ETL process due to resource constraints failing execution (#584) (a8ca0e0)
- Resolution to issue where Spend_HRK fields no longer exist in source file for google_political_ads data load process (#598) (09008dc)
- Resolve data file does not exist in Storms database pipeline. (#559) (ba0263b)
- Resolve data type change trip_id from integer to string in austin.bikeshare_trips dataset. (#607) (1020008)
- Resolve datatype switch for New York - station_id from int to guid (#557) (5daf41d)
- Resolve GKEPod pickup process (#556) (3d8ed98)
- Resolve issue with ordinal positioning of output file for loading FDA food enforcement data. (#614) (a3e2606)
- Resolve issues causing gsod_by_year pipeline to break in noaa dag. (#613) (8b7615f)
- Resolve missing references in pipeline.yaml Austin Bikeshare Trips. (#610) (a55d895)
- Resolve out of memory issues with Covid 19 symptom search dataset. (#568) (2ff1ff4)
- Resolve Resource Issue In Production Version Of Austin Pipeline (#528) (b2befa4)
- Resolve syntax error no 2 in pipeline.yaml for google political ads dataset. (#672) (8dd6fcc)
- Resolve syntax error no 3 in pipeline.yaml for google political ads. (#673) (395381f)
- Resolve Terraform Configs For Carbon-Free Energy calculator (#404) (b86a51d)
- Resolve variables for bucket and project in Open Buildings Dataset. (#465) (d916ad3)
- Resolved bad field data for field deleted in table full. (#827) (484ff4f)
- Resolved dtype error causing DAG failure for IMDB and adjusted memory and ephemeral storage appropriately. (#583) (350455a)
- Resolved issue in GSOD_BY_YEAR where files with 402 errors (#783) (08b1529)
- Resolved version increment auto-detect when updating City Health Dashboard dataset (#582) (1045874)
- Resolves lint errors in both noaa and thelook_ecommerce dataset deployments. (#617) (d4b0909)
- Resolving resource outage error. (#835) (13b315e)
- Rolling back to previous implementation. Modified for v2-2024 environment. (#777) (2f81461)
- Split EUMETSAT DAG to v3 and v4 tasks. (#589) (f386ccc)
This PR was generated with Release Please. See documentation.