etl icon indicating copy to clipboard operation
etl copied to clipboard

Remove v1 pipeline logic from data pipeline

Open stephen-soltesz opened this issue 2 years ago • 3 comments

Because the data pipeline co-existed with two versions v1 & v2, and now that the v1 data pipeline is decommissioned, we can begin deleting code relevant only to the v1 pipeline. This issue will track that effort.

  • [x] Delete Task queues in all projects.
  • [x] Remove functions
  • [x] Remove obsolete flags
  • [x] Remove dependencies on annotation-service repo
  • [x] Remove WEB-INF directory
  • [x] Remove Legacy/Annotation rows from Pipeline: overview dashboard
  • [x] Deploy all changes to production and verify no regressions

stephen-soltesz avatar Jun 02 '22 19:06 stephen-soltesz

https://github.com/m-lab/etl/releases/tag/prod-v2.21.0

stephen-soltesz avatar Jun 15 '22 17:06 stephen-soltesz

Complete for etl repo.

stephen-soltesz avatar Jun 15 '22 17:06 stephen-soltesz

This issue will track remove v1 logic from the gardener component also.

  • [x] Add go.mod support to gardener
  • [x] Unused flags and environment variable dependencies
  • [x] Eliminate (if obsolete) or consolidate datastore saver logic
  • [x] Eliminate unused state package (e.g. state.DatastoreSaver appears redundant)
  • [x] Eliminate unused cloud/bq/sanity.go logic
  • [x] Remove v1 client logic in client package and handler

stephen-soltesz avatar Jun 15 '22 17:06 stephen-soltesz