starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

Published 20 hours ago • verified publisher icon StarRocks

[WIP][Enhancement] support file sink in pipeline engine

Open silverbullet233 opened this issue 3 years ago • 7 comments

What type of PR is this：

  • [ ] BugFix
  • [x] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Which issues of this PR fixes ：

Fixes #

Problem Summary(Required) ：

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] I have added user document for my new feature or new function

silverbullet233 avatar Sep 19 '22 11:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 19 '22 11:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 20 '22 01:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 20 '22 02:09 silverbullet233

run starrocks_fe_unittest

silverbullet233 avatar Sep 20 '22 03:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 21 '22 02:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 26 '22 08:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 26 '22 09:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 27 '22 06:09 silverbullet233

run starrocks_fe_unittest

silverbullet233 avatar Sep 27 '22 06:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 27 '22 07:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 28 '22 07:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 28 '22 07:09 silverbullet233

run starrocks_fe_unittest

silverbullet233 avatar Sep 28 '22 08:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 28 '22 08:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 29 '22 02:09 silverbullet233

  1. write some tests for the core functions?
  2. any performance comparsion to the non-pipeline way?

fzhedu avatar Sep 30 '22 01:09 fzhedu

run starrocks_admit_test

silverbullet233 avatar Sep 30 '22 02:09 silverbullet233

  1. write some tests for the core functions?
  2. any performance comparsion to the non-pipeline way? @fzhedu
  1. daily test has some cases can cover it.
  2. whether it's in pipeline or non-pipeline engine, the main bottleneck is the sync io of serially writing files. I do a simple test in my env, SELECT * FROM tpch.lineitem limit 1000000 INTO OUTFILE "hdfs://xx" PROPERTIES ("broker.username" = "broker1", "column_separator" = ",","line_delimiter" = "\n","max_file_size" = "5MB"); and it takes about the same time in pipeline and non-pipeline engine.

silverbullet233 avatar Sep 30 '22 05:09 silverbullet233

run starrocks_admit_test

silverbullet233 avatar Sep 30 '22 05:09 silverbullet233

run starrocks_admit_test

wanpengfei-git avatar Oct 08 '22 06:10 wanpengfei-git

run starrocks_admit_test

wanpengfei-git avatar Oct 08 '22 11:10 wanpengfei-git

run starrocks_admit_test

wanpengfei-git avatar Oct 08 '22 11:10 wanpengfei-git

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

sonarqubecloud[bot] avatar Oct 08 '22 11:10 sonarqubecloud[bot]

[FE PR Coverage Check]

:heart_eyes: pass : 1 / 1 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/planner/ResultSink.java 1 1 100.00% []

wanpengfei-git avatar Oct 08 '22 11:10 wanpengfei-git