databend
databend copied to clipboard
copy from external location not work well with copied_files
Summary
when copy from 's3:://aa/bb/cc/' connection=(...), the root of tmp stage is aa/bb/cc/.
if there are 2 files
aa/bb/cc/x.txt
aa/bb/dd/x.txt
copy from 's3:://aa/bb/cc/' will record 'x.txt' as copied files, and when copy from 's3:://aa/bb/dd/' aa/bb/dd/x.txt will be skipped.
related code: https://github.com/datafuselabs/databend/blob/fbea9824fc3e3a3c6c7fd707cc15172cfad0cf28/src/query/sql/src/planner/binder/location.rs#L473
before this problem is solved, in production, it is recommended to use a named stage instead.