databend icon indicating copy to clipboard operation
databend copied to clipboard

feat: remove streaming upload?

Open youngsofun opened this issue 1 year ago • 1 comments

Summary

it is only used to load local files to Databend. instead user can

  1. upload to stage and run copy into table instead.
  2. if server runs locally only for test, user can copy into table from fs://

cons compare with copy:

  1. slow because of only one ONE stream can be used, and can not ultilize cluster (unless user run multi post by himself)
  2. for parquet files, can not read the meta at file end until.
  3. it is a insert, so no tracking of file paths like in copy.

youngsofun avatar Jun 03 '24 00:06 youngsofun

Yes, but it would be better to introduce a settings like enable_streaming_upload, default is false.

BohuTANG avatar Jun 03 '24 00:06 BohuTANG