doris icon indicating copy to clipboard operation
doris copied to clipboard

[fix](Export) fix some issues of Export

Open BePPPower opened this issue 9 months ago • 4 comments

Proposed changes

Issue Number: close #xxx

  1. Forbid rollback to the old optimizer in Export task. Since originStmt is empty, reverting to the old optimizer when the new optimizer is enabled is meaningless.

  2. Display parallelism in 'show export'.

  3. Create an initial Map for resultAttachedInfo to avoid NullPointerException.

  4. Remove the originStmt added in this PR #31020, because the Export statement is underlying the Outfile statement, it cannot be treated as OriginStmt.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

BePPPower avatar Apr 30 '24 07:04 BePPPower

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

doris-robot avatar Apr 30 '24 07:04 doris-robot

run buildall

BePPPower avatar Apr 30 '24 07:04 BePPPower

TPC-H: Total hot run time: 41319 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69d60829beabbd91463557728787d336af5c3e86, data reload: false

------ Round 1 ----------------------------------
q1	17593	4362	4307	4307
q2	2018	188	189	188
q3	10450	1308	1239	1239
q4	10190	831	796	796
q5	7493	2734	2748	2734
q6	246	134	134	134
q7	1038	631	598	598
q8	9235	2162	2110	2110
q9	9523	6748	6722	6722
q10	9301	3929	3929	3929
q11	445	252	257	252
q12	455	224	246	224
q13	18041	3159	3212	3159
q14	251	220	215	215
q15	519	464	475	464
q16	513	422	396	396
q17	1011	678	658	658
q18	8278	7724	7676	7676
q19	3280	1585	1555	1555
q20	634	351	334	334
q21	5183	4473	3355	3355
q22	352	274	288	274
Total cold run time: 116049 ms
Total hot run time: 41319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4537	4407	4370	4370
q2	369	281	267	267
q3	3168	2942	2884	2884
q4	2034	1595	1635	1595
q5	5377	5520	5457	5457
q6	215	124	124	124
q7	2359	1951	1966	1951
q8	3292	3444	3410	3410
q9	8639	8783	8813	8783
q10	4096	3734	3818	3734
q11	593	495	490	490
q12	807	631	624	624
q13	15986	3135	3128	3128
q14	290	270	278	270
q15	529	481	494	481
q16	471	429	456	429
q17	1791	1518	1517	1517
q18	7662	7508	7319	7319
q19	1639	1475	1524	1475
q20	1945	1784	1764	1764
q21	8468	4933	4909	4909
q22	575	496	508	496
Total cold run time: 74842 ms
Total hot run time: 55477 ms

doris-robot avatar Apr 30 '24 08:04 doris-robot

TPC-DS: Total hot run time: 186701 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69d60829beabbd91463557728787d336af5c3e86, data reload: false

query1	920	374	347	347
query2	6442	2310	2339	2310
query3	6642	213	216	213
query4	22967	21122	21206	21122
query5	4096	408	413	408
query6	271	186	184	184
query7	4575	285	297	285
query8	245	188	184	184
query9	8507	2432	2424	2424
query10	439	247	260	247
query11	14571	14184	14052	14052
query12	140	92	88	88
query13	1643	371	375	371
query14	10831	7936	7479	7479
query15	255	178	166	166
query16	8063	269	270	269
query17	1712	578	578	578
query18	2095	281	278	278
query19	289	162	153	153
query20	100	92	88	88
query21	201	128	124	124
query22	5068	4939	4863	4863
query23	33899	33196	33178	33178
query24	6524	2942	2824	2824
query25	585	385	380	380
query26	699	151	149	149
query27	1937	318	336	318
query28	4152	2072	2079	2072
query29	850	620	621	620
query30	250	162	153	153
query31	965	745	755	745
query32	91	55	58	55
query33	478	264	255	255
query34	872	476	478	476
query35	759	687	659	659
query36	1034	944	900	900
query37	109	68	67	67
query38	3144	2971	3006	2971
query39	1593	1524	1562	1524
query40	201	126	126	126
query41	45	41	42	41
query42	107	99	97	97
query43	582	548	539	539
query44	1055	734	739	734
query45	268	263	235	235
query46	1055	736	734	734
query47	1955	1875	1875	1875
query48	386	309	306	306
query49	930	388	395	388
query50	780	378	388	378
query51	6790	6784	6668	6668
query52	103	92	91	91
query53	358	295	287	287
query54	264	237	226	226
query55	79	71	72	71
query56	239	224	223	223
query57	1170	1157	1116	1116
query58	212	200	196	196
query59	3381	3238	3098	3098
query60	252	235	238	235
query61	107	90	88	88
query62	582	444	457	444
query63	302	278	279	278
query64	8280	7216	7201	7201
query65	3101	3043	3032	3032
query66	807	344	351	344
query67	15868	15474	15022	15022
query68	9558	558	556	556
query69	608	322	328	322
query70	1377	1081	1110	1081
query71	461	268	285	268
query72	8300	2553	2370	2370
query73	1604	331	337	331
query74	6489	6130	6031	6031
query75	4557	2650	2602	2602
query76	5382	906	1060	906
query77	704	266	265	265
query78	11180	10199	10084	10084
query79	12645	522	523	522
query80	2427	432	423	423
query81	503	241	218	218
query82	666	96	93	93
query83	211	161	162	161
query84	269	86	86	86
query85	1183	274	257	257
query86	398	296	279	279
query87	3289	3065	3099	3065
query88	5206	2429	2414	2414
query89	530	382	372	372
query90	2382	187	182	182
query91	129	94	97	94
query92	57	46	47	46
query93	7347	517	513	513
query94	1496	189	176	176
query95	1095	1088	1088	1088
query96	599	266	268	266
query97	3113	2917	2918	2917
query98	229	224	217	217
query99	1242	886	844	844
Total cold run time: 302508 ms
Total hot run time: 186701 ms

doris-robot avatar Apr 30 '24 08:04 doris-robot

run buildall

BePPPower avatar May 08 '24 03:05 BePPPower

run buildall

BePPPower avatar May 08 '24 06:05 BePPPower

run buildall

BePPPower avatar May 13 '24 13:05 BePPPower

TPC-H: Total hot run time: 40939 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4e317f09ec573819911effb700829ec3c20db41, data reload: false

------ Round 1 ----------------------------------
q1	17618	4312	4725	4312
q2	2021	187	191	187
q3	10984	1226	1216	1216
q4	10769	833	905	833
q5	7487	2723	2666	2666
q6	223	133	136	133
q7	1014	615	589	589
q8	9576	2120	2103	2103
q9	9207	6651	6595	6595
q10	9022	3697	3688	3688
q11	484	239	231	231
q12	399	224	217	217
q13	18553	2934	2945	2934
q14	255	228	222	222
q15	526	485	471	471
q16	514	377	381	377
q17	958	710	747	710
q18	8031	7460	7520	7460
q19	5217	1491	1467	1467
q20	687	310	301	301
q21	5122	3945	3968	3945
q22	360	282	311	282
Total cold run time: 119027 ms
Total hot run time: 40939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4266	4218	4218
q2	357	263	263	263
q3	2979	2757	2694	2694
q4	1871	1623	1618	1618
q5	5289	5266	5268	5266
q6	207	124	125	124
q7	2263	1902	1872	1872
q8	3193	3325	3331	3325
q9	8300	8303	8322	8303
q10	3893	3682	3645	3645
q11	573	490	502	490
q12	747	586	592	586
q13	16524	2920	2946	2920
q14	299	279	247	247
q15	514	472	453	453
q16	472	418	417	417
q17	1772	1534	1470	1470
q18	7578	7513	7438	7438
q19	1646	1553	1511	1511
q20	1935	1732	1775	1732
q21	5147	4980	4888	4888
q22	575	479	489	479
Total cold run time: 70463 ms
Total hot run time: 53959 ms

doris-robot avatar May 13 '24 13:05 doris-robot

TPC-DS: Total hot run time: 187156 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4e317f09ec573819911effb700829ec3c20db41, data reload: false

query1	914	358	348	348
query2	6446	2384	2230	2230
query3	6659	206	214	206
query4	23158	21223	21457	21223
query5	4163	421	436	421
query6	264	172	176	172
query7	4583	300	286	286
query8	244	187	185	185
query9	8590	2413	2413	2413
query10	442	253	277	253
query11	14712	14258	14071	14071
query12	147	93	88	88
query13	1638	379	381	379
query14	10132	8439	8528	8439
query15	268	166	168	166
query16	8153	258	265	258
query17	1874	558	544	544
query18	2176	295	267	267
query19	223	149	150	149
query20	92	87	83	83
query21	200	129	128	128
query22	5020	4896	4770	4770
query23	34165	33503	33444	33444
query24	11925	2940	2885	2885
query25	643	367	362	362
query26	1746	158	155	155
query27	2946	319	322	319
query28	7572	2053	2039	2039
query29	1002	619	603	603
query30	299	146	152	146
query31	958	742	744	742
query32	93	59	55	55
query33	746	265	258	258
query34	997	469	476	469
query35	790	694	666	666
query36	1061	884	927	884
query37	184	68	68	68
query38	2883	2752	2734	2734
query39	1646	1614	1557	1557
query40	275	123	129	123
query41	40	39	38	38
query42	108	95	97	95
query43	581	535	529	529
query44	1215	724	739	724
query45	269	225	254	225
query46	1079	703	747	703
query47	1943	1845	1886	1845
query48	359	299	300	299
query49	1194	388	404	388
query50	774	385	389	385
query51	6815	6653	6741	6653
query52	106	91	92	91
query53	346	277	288	277
query54	1018	422	434	422
query55	79	73	73	73
query56	233	223	226	223
query57	1230	1145	1126	1126
query58	219	207	206	206
query59	3413	3145	3308	3145
query60	251	240	239	239
query61	96	89	89	89
query62	670	462	471	462
query63	319	284	285	284
query64	9726	7389	7356	7356
query65	3162	3110	3099	3099
query66	1393	344	344	344
query67	15183	14778	14820	14778
query68	5492	549	537	537
query69	541	290	294	290
query70	1120	1159	1148	1148
query71	448	266	273	266
query72	7748	2550	2361	2361
query73	711	321	323	321
query74	6560	6113	6059	6059
query75	3722	2578	2614	2578
query76	3257	931	936	931
query77	603	267	258	258
query78	10624	10206	10042	10042
query79	2217	512	516	512
query80	1198	448	443	443
query81	511	217	218	217
query82	698	93	93	93
query83	195	164	165	164
query84	265	89	88	88
query85	1439	276	324	276
query86	454	327	313	313
query87	3300	3114	3114	3114
query88	3929	2327	2335	2327
query89	460	381	382	381
query90	1934	192	186	186
query91	127	99	95	95
query92	57	52	52	52
query93	2032	520	513	513
query94	1148	186	184	184
query95	401	304	300	300
query96	586	270	264	264
query97	3198	3018	3012	3012
query98	252	219	223	219
query99	1320	926	939	926
Total cold run time: 288397 ms
Total hot run time: 187156 ms

doris-robot avatar May 13 '24 14:05 doris-robot

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar May 14 '24 09:05 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar May 14 '24 09:05 github-actions[bot]