artefactory-connectors-kit icon indicating copy to clipboard operation
artefactory-connectors-kit copied to clipboard

As a User, I can provide all arguments as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer

Open gabrielleberanger opened this issue 4 years ago • 0 comments

WHY The following readers/writers are using implicit config.<VAR> variables:

  • gcs_reader, via the object_storage_reader
    • config.PROJECT_ID
  • s3_reader, via the object_storage_reader
    • config.REGION_NAME, config.AWS_ACCESS_KEY_ID, config.AWS_SECRET_ACCESS_KEY
  • gcs_writer
    • config.PROJECT_ID
  • bq_writer
    • config.PROJECT_ID

These variables are not provided by the user as click options, but as environment variables. These environment variables are retrieved from the following snippet that appears in the nck/config.py module:

for key, var in os.environ.items():
    locals()[key] = var

This can be quite confusing for the user, and does not match the common NCK logic.

HOW Explicitly implement these variables as click options

gabrielleberanger avatar Feb 08 '21 10:02 gabrielleberanger