databend icon indicating copy to clipboard operation
databend copied to clipboard

copy from external location not work well with copied_files

Open youngsofun opened this issue 1 year ago • 0 comments

Summary

when copy from 's3:://aa/bb/cc/' connection=(...), the root of tmp stage is aa/bb/cc/.

if there are 2 files

aa/bb/cc/x.txt
aa/bb/dd/x.txt

copy from 's3:://aa/bb/cc/' will record 'x.txt' as copied files, and when copy from 's3:://aa/bb/dd/' aa/bb/dd/x.txt will be skipped.

related code: https://github.com/datafuselabs/databend/blob/fbea9824fc3e3a3c6c7fd707cc15172cfad0cf28/src/query/sql/src/planner/binder/location.rs#L473

before this problem is solved, in production, it is recommended to use a named stage instead.

youngsofun avatar Jun 18 '24 01:06 youngsofun