snowplow-bigquery-loader icon indicating copy to clipboard operation
snowplow-bigquery-loader copied to clipboard

Loader: make behaviour consistent with Beam Enrich

Open simplylizz opened this issue 4 years ago • 0 comments

  1. Different arg name for dataflow jobs: --job-name for enricher vs --jobName for the loader.
  2. Enricher fails if a job name isn't specified, the loader generates it (I think the first is preferred since they use the same input/output resources and conflicts here may have an unpredictable effect; also job-name is unique within dataflow project and allows you to update your pipelines).
  3. Enricher exits after launching a pipeline while the loader hangs and outputs some info into console.

I think it would be nice to have consistent behavior.

simplylizz avatar Jun 30 '20 21:06 simplylizz