artefactory-connectors-kit
artefactory-connectors-kit copied to clipboard
artefactory
ACK is an E(T)L tool specialized in API data ingestion. It is accessible through a Command-Line Interface. The application allows you to easily extract, stream and load data (with minimum transformati...
To view the ui, run ` streamlit run ack/ui/ui-streamlit.py` Make sure you have checked _all_ steps below. ### Issue - [ ] My PR partially adresses #72 - [ ]...
WHY The following readers/writers are using implicit `config.` variables: - `gcs_reader`, via the `object_storage_reader` - `config.PROJECT_ID` - `s3_reader`, via the `object_storage_reader` - `config.REGION_NAME`, `config.AWS_ACCESS_KEY_ID`, `config.AWS_SECRET_ACCESS_KEY` - `gcs_writer` - `config.PROJECT_ID` -...
### Issue My PR resolves #93 ### Description - change all environment arguments to Click options and json args in config.py files ### Documentation - The documentation has been adapted...
Here is the first implementation of a generique bytes stream. You can see the end to end test of it in test/end_to_end, which test writing from and to S3.
WHY Currently, we don't have tests that verify the end-to-end workflow because we can't share credentials from real customers to every contributor. So, it might be a good idea to...
WHY Today, each reader uses its own authentication method: either a client class or a helper method. Available client classes are available under the `nck/clients` directory (after the refacto, clients...
WHY We currently only have a technical documentation which is good but it could be interesting to have a one pager or a short deck which explains the value the...
WHY The solution proposed by the [Airbyte](https://airbyte.io/) start-up is very close to what we had in mind for NCK (open-source application, EL(T) approach, similar data sources, configuration through an UI...
WHY Today, the only output stream format available is `.njson` (i.e. a file with n lines, each line being a dictionnary). This format has two downsides: - It does not...
WHY Today, the behavior of readers expecting date parameters (`start_date`, `end_date`, `date_range`, etc.) is not harmonized. In particular, readers usually do not know how to prioritize the date parameters given...
Metadata
Owner
Metadata
ACK is an E(T)L tool specialized in API data ingestion. It is accessible through a Command-Line Interface. The application allows you to easily extract, stream and load data (with minimum transformati...