astro-sdk
astro-sdk copied to clipboard
Include File size and File type in native path selection logic
trafficstars
Please describe the feature you'd like to see Currently, we are only using Source and Destination to determine the transfer logic to be executed - default path / native path. There are issues with this approach
- File Types - We are assuming the native transfer is supported for all file types. This is not the case with Snowflake and Bigquey.
- File Size - When transferring data from the S3 to Bigquery, we create a Data Transfer Job in Biquery. Which is overkill for the smaller files.
Describe the solution you'd like When we include the File Type and File Size in the decision to select a native or default path. We can better select the path which can lead to a better user experience and faster transfer.
Acceptance Criteria
- [ ] All checks and tests in the CI should pass
- [ ] Unit tests (90% code coverage or more, once available)
- [ ] Integration tests (if the feature relates to a new database or external service)
- [ ] Example DAG
- [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- [ ] Exception handling in case of errors
- [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
- [ ] Improve the documentation (README, Sphinx, and any other relevant)
- [ ] How to use Guide for the feature (example)