Clean up unit tests for onestop-python-client

[kenhtanaka]

Summary

As a	Developer
I want to be able to	Change all constructor signatures to not take a config file path but parameters and improve unit tests for the onestop-python-client module
So that I can	have better test coverage without errors and not pass a config file into every constructor.

• [x] Change classes so none have config files passed in, only parameters needed.
• [x] Updated effected documentation
• [x] Get unit tests working
• [x] Change unittests to not load config but have their own dict they pass in, no need for loading from a config file for unittests.
• [x] Move unit tests to onestop-python-client/tests/unit and put test_WebPublisher in onestop-python-client/tests/integration
• [x] Update Circle CI to run all tests, had to rename/move unit tests to test/unit/test*.py so python unittest discovery can find them. Moved integration test to test/integration
• [x] Set build environment variables for integration tests to use. ~- [ ] Update scripts to at least accurate, unsure if all can work.~ Moved to new story
• [x] Refactored S3Utils.connect so clearer, adds one more param but is clearer.

[erinreeves]

Questions:

• Lots of merged but not deleted branches, so unsure if should look at any for if these fixes have been done.
• Future story: Should we refactor out non-extractor code from Extractor class? Such as the connecting to S3?
• Refactor Chris did puts the parsing of config files on whatever calls S3MessageAdapter, S3Utils, SqsConsumer. Reflected in current script s3_notification_handler. Verification ok?
• Someone give me an example of executing python3 scripts/sqs-to-registry/s3_notification_handler.py?

[erinreeves]

Also need find story about updating circleCI config to remove CLI build.

[erinreeves]

Only have log_level in one config?

[erinreeves]

Check if relevant for story 1233: Follow upstory for S3Utils:

1. Rename method def get_csv_s3(self, boto_client, bucket, key): https://github.com/cedardevs/onestop-clients/blob/master/onestop-python-client/onestop/util/S3Utils.py#L99 so directly below we have read_bytes_s3 ... so maybe we have get_s3_bytes get_s3_file read_bytes_s3 -> get_s3_bytes get_csv_s3 -> get_s3_file

2. For CsbExtractor, shouldn't be doing any connecting https://github.com/cedardevs/onestop-clients/blob/master/onestop-python-client/onestop/extract/CsbExtractor.py#L42

[erinreeves]

S3MessageAdapter:

• look into transform method used to return im_message.file_format, headers, - gone now. why?

[erinreeves]

Classes todo modification:

• [x] S3Utils

• [x] CsbExtractor

• [x] WebPublisher

• [x] KafkaConsumer

• [x] KafkaPublisher

• [x] SqsConsumer

• [x] Change unittests to not load config but have their own dict they pass in, no need for loading from a config file for unittests.

[erinreeves]

What is the incoming collection_uuid in KafkaPublisher class for publish_collection and publish_granule?

[erinreeves]

Build passing!!

[erinreeves]

Which scripts can be run locally versus have to be copied to a pod to be executed?

archive_client_integration.py bucket_automation.py launch_delete_handler.py launch_e2e.py launch_kafka_publisher.py launch_pyconsumer.py

sme/ sme.py smeFunc.py spatial.py

sqs-to-registry/ s3_notification_handler.py

[erinreeves]

https://github.com/cedardevs/onestop-clients/pull/57