doris icon indicating copy to clipboard operation
doris copied to clipboard

[enhancement](broker) broker load support tencent cos

Open nextdreamblue opened this issue 2 years ago • 1 comments

Signed-off-by: nextdreamblue [email protected]

Proposed changes

Issue Number: close #xxx

Problem summary

broker load support tencent cos.

Checklist(Required)

  1. Does it affect the original behavior:
    • [ ] No
  2. Has unit tests been added:
    • [ ] No Need
  3. Has document been added or modified:
    • [ ] Yes
  4. Does it need to update dependencies:
    • [ ] No
  5. Are there any changes that cannot be rolled back:
    • [ ] No

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

nextdreamblue avatar Sep 21 '22 03:09 nextdreamblue

Why not access cos with s3?

morningman avatar Sep 22 '22 01:09 morningman

Why not access cos with s3?

使用腾讯cos对象存储的协议去访问cos相比原生s3a协议的优势:

  1. 比如针对对象存储的数据读取,又进行数据的预读操作
  2. 针对文件路径的查询操作,在判断目录是否是目录时,有进行list-1的优化操作
  3. 支持了对象存储这边的融合桶支持
  4. 支持了ranger的访问控制协议

  5. 其他兼容性优化

nextdreamblue avatar Oct 26 '22 02:10 nextdreamblue

@morningman please review, thanks

nextdreamblue avatar Oct 26 '22 12:10 nextdreamblue

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.75 seconds load time: 434 seconds storage size: 17151837064 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221122085621_clickbench_pr_50573.html

hello-stephen avatar Oct 28 '22 10:10 hello-stephen

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Nov 04 '22 06:11 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Nov 04 '22 06:11 github-actions[bot]

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Nov 17 '22 02:11 github-actions[bot]

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Nov 22 '22 10:11 github-actions[bot]