FlowKit icon indicating copy to clipboard operation
FlowKit copied to clipboard

Add a `FileExistsSensor` before creating the foreign table, using `pg_stat_file` or similar

Open greenape opened this issue 1 year ago • 1 comments

          The approach makes sense. Although this still leaves us in the situation of using the fail/retry mechanism in lieu of a sensor; it's just that it will now be the `CreateForeignStagingTableOperator` task that fails and retries until the file appears, rather than the "wait for data" task. I wonder whether we could add a `FileExistsSensor` before creating the foreign table, using `pg_stat_file` or similar?

Originally posted by @jc-harrison in https://github.com/Flowminder/FlowKit/pull/6448#pullrequestreview-1891105882

greenape avatar Feb 21 '24 09:02 greenape

My suggestion from that thread is that we do something using pg_ls_dir to allow for checking for multiple files.

greenape avatar Feb 21 '24 09:02 greenape