doris icon indicating copy to clipboard operation
doris copied to clipboard

[enhance](mtmv)Mv refresh on commit

Open zddr opened this issue 9 months ago • 16 comments

Proposed changes

Issue Number: close #xxx

support refresh MTMV when base table data change / drop partition/replace partition

CREATE MATERIALIZED VIEW mv1
        REFRESH ON COMMIT
        AS 
        SELECT xxx;

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

zddr avatar May 08 '24 10:05 zddr

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

doris-robot avatar May 08 '24 10:05 doris-robot

run buildall

zddr avatar May 13 '24 02:05 zddr

TPC-H: Total hot run time: 41875 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f6889fe71df769fad15c409eb9a317277a86c4f, data reload: false

------ Round 1 ----------------------------------
q1	17603	4380	4274	4274
q2	2026	180	185	180
q3	10476	1192	1217	1192
q4	10192	827	776	776
q5	7476	2708	2748	2708
q6	226	133	133	133
q7	1038	619	585	585
q8	9219	2112	2071	2071
q9	9361	6720	6658	6658
q10	9628	3934	3852	3852
q11	457	234	237	234
q12	515	258	222	222
q13	17181	3211	3274	3211
q14	250	211	214	211
q15	512	480	494	480
q16	485	397	404	397
q17	969	727	726	726
q18	8366	7743	7704	7704
q19	4682	1554	1529	1529
q20	654	317	317	317
q21	5227	4135	4227	4135
q22	373	280	289	280
Total cold run time: 116916 ms
Total hot run time: 41875 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4580	4413	4370	4370
q2	380	284	281	281
q3	3101	2941	2780	2780
q4	1885	1600	1582	1582
q5	5553	5557	5622	5557
q6	208	124	125	124
q7	2381	2005	2026	2005
q8	3308	3400	3351	3351
q9	8600	8657	8647	8647
q10	3934	3749	3860	3749
q11	578	487	490	487
q12	788	621	637	621
q13	16760	3088	3147	3088
q14	312	267	280	267
q15	513	479	482	479
q16	507	423	436	423
q17	1748	1490	1458	1458
q18	7665	7538	7507	7507
q19	1635	1533	1496	1496
q20	1963	1764	1752	1752
q21	5002	4780	4889	4780
q22	591	485	470	470
Total cold run time: 71992 ms
Total hot run time: 55274 ms

doris-robot avatar May 13 '24 02:05 doris-robot

TPC-DS: Total hot run time: 186310 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f6889fe71df769fad15c409eb9a317277a86c4f, data reload: false

query1	904	360	347	347
query2	6485	2439	2370	2370
query3	6642	214	216	214
query4	24404	21249	21099	21099
query5	4146	404	410	404
query6	260	179	170	170
query7	4586	294	282	282
query8	243	183	191	183
query9	8640	2425	2413	2413
query10	439	250	265	250
query11	14846	14162	14194	14162
query12	136	89	85	85
query13	1632	359	371	359
query14	10027	8331	7121	7121
query15	223	174	170	170
query16	7867	259	254	254
query17	1759	545	537	537
query18	1896	275	272	272
query19	203	149	145	145
query20	89	86	84	84
query21	192	133	127	127
query22	5132	4926	4851	4851
query23	34116	33482	33403	33403
query24	10644	2943	2899	2899
query25	564	359	359	359
query26	699	156	151	151
query27	2210	321	314	314
query28	5915	2069	2057	2057
query29	870	625	606	606
query30	260	158	159	158
query31	1016	764	737	737
query32	83	50	52	50
query33	626	255	243	243
query34	869	484	489	484
query35	779	660	652	652
query36	1117	959	949	949
query37	108	66	65	65
query38	3039	2787	2779	2779
query39	1613	1548	1538	1538
query40	191	119	128	119
query41	40	38	37	37
query42	106	96	93	93
query43	548	563	519	519
query44	1117	720	730	720
query45	269	252	246	246
query46	1076	722	690	690
query47	1987	1918	1914	1914
query48	373	305	292	292
query49	844	389	387	387
query50	773	392	405	392
query51	6698	6736	6616	6616
query52	95	90	88	88
query53	345	278	275	275
query54	850	427	433	427
query55	72	72	70	70
query56	247	219	224	219
query57	1211	1175	1174	1174
query58	215	199	198	198
query59	3563	3349	3433	3349
query60	261	227	233	227
query61	89	87	98	87
query62	645	473	491	473
query63	307	271	277	271
query64	8446	7381	7359	7359
query65	3107	3064	3064	3064
query66	784	352	337	337
query67	15386	14972	14995	14972
query68	4519	526	532	526
query69	463	305	302	302
query70	1149	1126	1162	1126
query71	355	276	270	270
query72	7096	2549	2387	2387
query73	698	328	335	328
query74	6435	6149	6042	6042
query75	3318	2580	2611	2580
query76	2195	946	989	946
query77	375	263	263	263
query78	10663	10105	10012	10012
query79	1971	503	503	503
query80	1112	429	421	421
query81	545	216	223	216
query82	945	91	91	91
query83	241	160	161	160
query84	252	87	86	86
query85	1272	347	256	256
query86	448	332	303	303
query87	3255	3153	3124	3124
query88	3307	2425	2430	2425
query89	461	373	384	373
query90	2004	185	180	180
query91	118	96	98	96
query92	54	46	45	45
query93	1902	505	491	491
query94	1230	180	176	176
query95	392	295	299	295
query96	590	274	265	265
query97	3169	2972	3055	2972
query98	236	222	219	219
query99	1207	873	874	873
Total cold run time: 276901 ms
Total hot run time: 186310 ms

doris-robot avatar May 13 '24 02:05 doris-robot

run buildall

zddr avatar May 16 '24 06:05 zddr

run buildall

zddr avatar May 16 '24 07:05 zddr

run buildall

zddr avatar May 16 '24 11:05 zddr

run buildall

zddr avatar May 16 '24 13:05 zddr

run p0

zddr avatar May 16 '24 23:05 zddr

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar May 17 '24 08:05 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar May 17 '24 08:05 github-actions[bot]