astro-sdk
astro-sdk copied to clipboard
Append operator should tolerate non-existent target tables
Please describe the feature you'd like to see It would be nice if the append operator creates the target table if it does not already exist. I am building a pipeline where the destination table does not exist on the first run, but will exist on subsequent runs. I could add a task at the beginning that ensures the target table exists, but it would be really cool if I didn't have to!
I can't think of many (any?) cases where I'd want to append to a table but only if it exists. I think I want "append" to functionally work how >> does in a shell - it will not complain if the target file doesn't exist. But I could be alone here!
Describe the solution you'd like
The append operator should attempt to create the target table if it doesn't exist, either by default or with some sort of create_target_table=True flag.
Are there any alternatives to this feature? We could better catch and report the error that results when the table doesn't exist, which for me was a KeyError deep inside sqlalchemy.
Acceptance Criteria
- [ ] All checks and tests in the CI should pass
- [ ] Unit tests (90% code coverage or more, once available)
- [ ] Integration tests (if the feature relates to a new database or external service)
- [ ] Example DAG
- [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- [ ] Exception handling in case of errors
- [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
- [ ] Improve the documentation (README, Sphinx, and any other relevant)
- [ ] How to use Guide for the feature (example)