starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Optimize ingestion performance for table with materialized index STEP 1

Open meegoo opened this issue 3 years ago • 9 comments

What type of PR is this:

  • [ ] bug
  • [ ] feature
  • [x] enhancement
  • [ ] refactor
  • [ ] others

Which issues of this PR fixes :

Fixes #7717

Problem Summary(Required) :

When a table has multiple materialized views, StarRocks will send the chunks to the tablets corresponding to the materialized views, which leads to the enlargement of network transmission bandwidth and unnecessary data serialization and transmission overhead. Since the materialized view and the base table use the same partition column and distribute in the same node when using colocate mv #7451, we can reuse the chunk data of the base table.

materialized index num main branch optimize branch SpeedUp
0 1314MB/s 1310MB/s
1 780MB/s 1274Mb/s 1.6x
3 420MB/s 1241MB/s 2.9x
5 286MB/s 1213MB/s 4.2x

using github_events data, 16 parallel 3 BE with 1.5GB/s network bandwidth create two columns of materialized view; materialized view computation is relatively small; ensure the bottleneck is network bandwidth and serialization.

NOTE We will enable it after colocate mv index implementation.

meegoo avatar Jun 10 '22 02:06 meegoo

[FE PR Coverage check]

:heart_eyes: pass : 0 / 0 (0%)

wanpengfei-git avatar Jun 16 '22 08:06 wanpengfei-git

run starrocks_be_unittest

meegoo avatar Jul 25 '22 03:07 meegoo

run starrocks_be_unittest

meegoo avatar Jul 25 '22 03:07 meegoo

run starrocks_fe_unittest

meegoo avatar Jul 28 '22 03:07 meegoo

@mergify rebase

meegoo avatar Aug 12 '22 02:08 meegoo

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Aug 12 '22 02:08 mergify[bot]

run starrocks_fe_unittest

meegoo avatar Aug 12 '22 03:08 meegoo

run starrocks_fe_unittest

meegoo avatar Aug 12 '22 03:08 meegoo

@mergify rebase

meegoo avatar Aug 15 '22 06:08 meegoo

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Aug 15 '22 06:08 mergify[bot]

LGTM for the LoadChannel and TabletsChannel parts.

sduzh avatar Aug 15 '22 06:08 sduzh

run starrocks_admit_test

wanpengfei-git avatar Aug 16 '22 06:08 wanpengfei-git

run starrocks_admit_test

meegoo avatar Aug 16 '22 06:08 meegoo

@mergify rebase

meegoo avatar Aug 16 '22 08:08 meegoo

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Aug 16 '22 08:08 mergify[bot]

run starrocks_admit_test

wanpengfei-git avatar Aug 16 '22 08:08 wanpengfei-git

run starrocks_fe_unittest

meegoo avatar Aug 16 '22 09:08 meegoo

run starrocks_admit_test

meegoo avatar Aug 16 '22 10:08 meegoo

run starrocks_fe_unittest

meegoo avatar Aug 16 '22 11:08 meegoo

run starrocks_fe_unittest

meegoo avatar Aug 16 '22 13:08 meegoo

run starrocks_fe_unittest

meegoo avatar Aug 16 '22 14:08 meegoo

run starrocks_fe_unittest

meegoo avatar Aug 16 '22 14:08 meegoo

[FE PR Coverage Check]

:heart_eyes: pass : 0 / 0 (0%)

wanpengfei-git avatar Aug 16 '22 15:08 wanpengfei-git