artefactory-connectors-kit
artefactory-connectors-kit copied to clipboard
As a User, I can provide all arguments as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer
WHY
The following readers/writers are using implicit config.<VAR>
variables:
-
gcs_reader
, via theobject_storage_reader
-
config.PROJECT_ID
-
-
s3_reader
, via theobject_storage_reader
-
config.REGION_NAME
,config.AWS_ACCESS_KEY_ID
,config.AWS_SECRET_ACCESS_KEY
-
-
gcs_writer
-
config.PROJECT_ID
-
-
bq_writer
-
config.PROJECT_ID
-
These variables are not provided by the user as click options, but as environment variables. These environment variables are retrieved from the following snippet that appears in the nck/config.py
module:
for key, var in os.environ.items():
locals()[key] = var
This can be quite confusing for the user, and does not match the common NCK logic.
HOW Explicitly implement these variables as click options