FlowKit
FlowKit copied to clipboard
Add a `FileExistsSensor` before creating the foreign table, using `pg_stat_file` or similar
The approach makes sense. Although this still leaves us in the situation of using the fail/retry mechanism in lieu of a sensor; it's just that it will now be the `CreateForeignStagingTableOperator` task that fails and retries until the file appears, rather than the "wait for data" task. I wonder whether we could add a `FileExistsSensor` before creating the foreign table, using `pg_stat_file` or similar?
Originally posted by @jc-harrison in https://github.com/Flowminder/FlowKit/pull/6448#pullrequestreview-1891105882
My suggestion from that thread is that we do something using pg_ls_dir to allow for checking for multiple files.