doris icon indicating copy to clipboard operation
doris copied to clipboard

[Enhancement] add information_schema.table_options(#32572)

Open Vallishp opened this issue 10 months ago • 38 comments

Proposed changes

Issue Number: close #32572

Support table_options in information_schema. UT result: mysql> select * from information_schema.table_options where table_schema="test_table_options_db"\G *************************** 1. row *************************** TABLE_NAME: rangetable TABLE_CATALOG: internal TABLE_SCHEMA: test_table_options_db DISTRIBUTE_KEY: user_id DISTRIBUTE_TYPE: HASH BUCKETS_NUM: 8 PARTITION_NUM: 3 PROPERTIES: min_load_replica_num = -1, data_sort.col_num = 6, group_commit_interval_ms = 10000, data_sort.sort_type = LEXICAL, is_being_synced = false, binlog.enable = false, binlog.ttl_seconds = 86400, inverted_index_storage_format = V1, time_series_compaction_empty_rowsets_threshold = 5, default.replication_allocation = tag.location.default: 1, time_series_compaction_level_threshold = 1, time_series_compaction_time_threshold_seconds = 3600, storage_format = V2, store_row_column = false, light_schema_change = true, enable_unique_key_merge_on_write = false, in_memory = false, file_cache_ttl_seconds = 0, group_commit_data_bytes = 134217728, compaction_policy = size_based, _auto_bucket = false, binlog.max_history_nums = 9223372036854775807, time_series_compaction_file_count_threshold = 2000, skip_write_index_on_load = false, disable_auto_compaction = false, time_series_compaction_goal_size_mbytes = 1024, storage_medium = HDD, enable_single_replica_compaction = false, compression = LZ4F, binlog.max_bytes = 9223372036854775807 *************************** 2. row *************************** TABLE_NAME: listtable TABLE_CATALOG: internal TABLE_SCHEMA: test_table_options_db DISTRIBUTE_KEY: user_id DISTRIBUTE_TYPE: HASH BUCKETS_NUM: 16 PARTITION_NUM: 3 PROPERTIES: min_load_replica_num = -1, data_sort.col_num = 6, group_commit_interval_ms = 10000, data_sort.sort_type = LEXICAL, is_being_synced = false, binlog.enable = false, binlog.ttl_seconds = 86400, inverted_index_storage_format = V1, time_series_compaction_empty_rowsets_threshold = 5, default.replication_allocation = tag.location.default: 1, time_series_compaction_level_threshold = 1, time_series_compaction_time_threshold_seconds = 3600, storage_format = V2, store_row_column = false, light_schema_change = true, enable_unique_key_merge_on_write = false, in_memory = false, file_cache_ttl_seconds = 0, group_commit_data_bytes = 134217728, compaction_policy = size_based, _auto_bucket = false, binlog.max_history_nums = 9223372036854775807, time_series_compaction_file_count_threshold = 2000, skip_write_index_on_load = false, disable_auto_compaction = false, time_series_compaction_goal_size_mbytes = 1024, storage_medium = HDD, enable_single_replica_compaction = false, compression = LZ4F, binlog.max_bytes = 9223372036854775807 *************************** 3. row *************************** TABLE_NAME: randomtable TABLE_CATALOG: internal TABLE_SCHEMA: test_table_options_db DISTRIBUTE_KEY: RANDOM DISTRIBUTE_TYPE: RANDOM BUCKETS_NUM: 16 PARTITION_NUM: 1 PROPERTIES: min_load_replica_num = -1, data_sort.col_num = 3, group_commit_interval_ms = 10000, data_sort.sort_type = LEXICAL, is_being_synced = false, binlog.enable = false, binlog.ttl_seconds = 86400, inverted_index_storage_format = V1, time_series_compaction_empty_rowsets_threshold = 5, default.replication_allocation = tag.location.default: 1, time_series_compaction_level_threshold = 1, time_series_compaction_time_threshold_seconds = 3600, storage_format = V2, store_row_column = false, light_schema_change = true, enable_unique_key_merge_on_write = false, in_memory = false, file_cache_ttl_seconds = 0, group_commit_data_bytes = 134217728, compaction_policy = size_based, _auto_bucket = false, binlog.max_history_nums = 9223372036854775807, time_series_compaction_file_count_threshold = 2000, skip_write_index_on_load = false, disable_auto_compaction = false, time_series_compaction_goal_size_mbytes = 1024, storage_medium = HDD, enable_single_replica_compaction = false, compression = LZ4F, binlog.max_bytes = 9223372036854775807 3 rows in set (0.03 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Vallishp avatar May 02 '24 13:05 Vallishp

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

doris-robot avatar May 02 '24 13:05 doris-robot

run buildall

Vallishp avatar May 02 '24 13:05 Vallishp

TPC-DS: Total hot run time: 185670 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8318a3f1939e51aa1301849e6628a624a0c099b1, data reload: false

query1	895	355	346	346
query2	6360	2425	2373	2373
query3	6640	205	207	205
query4	23169	21858	21791	21791
query5	4033	434	417	417
query6	265	182	176	176
query7	4541	305	301	301
query8	241	193	186	186
query9	8493	2363	2361	2361
query10	432	241	255	241
query11	15261	14756	14821	14756
query12	116	87	84	84
query13	1662	378	366	366
query14	10841	8387	6487	6487
query15	251	167	164	164
query16	8206	275	260	260
query17	1766	571	559	559
query18	2113	291	284	284
query19	322	155	146	146
query20	90	85	83	83
query21	197	127	121	121
query22	5036	4865	4911	4865
query23	33815	32769	33198	32769
query24	11459	2965	2893	2893
query25	645	379	372	372
query26	1573	158	150	150
query27	2883	320	311	311
query28	7376	2004	1996	1996
query29	973	682	605	605
query30	242	152	155	152
query31	945	715	724	715
query32	97	55	55	55
query33	759	238	232	232
query34	1065	478	467	467
query35	823	648	658	648
query36	1056	897	893	893
query37	124	66	62	62
query38	3189	3006	3013	3006
query39	1583	1556	1518	1518
query40	235	129	125	125
query41	39	35	35	35
query42	100	92	94	92
query43	557	560	535	535
query44	1227	732	746	732
query45	252	250	245	245
query46	1088	713	743	713
query47	1952	1854	1860	1854
query48	365	292	305	292
query49	917	383	377	377
query50	762	375	390	375
query51	6838	6674	6735	6674
query52	97	88	91	88
query53	345	286	288	286
query54	291	228	240	228
query55	78	72	71	71
query56	237	227	217	217
query57	1230	1117	1148	1117
query58	219	192	195	192
query59	3507	3098	3136	3098
query60	258	229	233	229
query61	88	85	88	85
query62	668	462	458	458
query63	304	289	281	281
query64	9436	7186	7266	7186
query65	3108	3068	3045	3045
query66	1355	340	336	336
query67	15701	14758	14916	14758
query68	9553	541	548	541
query69	607	321	304	304
query70	1330	1053	1091	1053
query71	535	270	269	269
query72	8539	2504	2360	2360
query73	1557	318	319	318
query74	6600	6079	6052	6052
query75	5365	2675	2623	2623
query76	5910	1034	1006	1006
query77	764	261	266	261
query78	11278	10311	10222	10222
query79	12500	520	504	504
query80	1459	425	432	425
query81	488	221	217	217
query82	281	88	91	88
query83	199	165	163	163
query84	259	85	83	83
query85	1015	309	264	264
query86	353	298	312	298
query87	3257	3094	3103	3094
query88	5065	2427	2426	2426
query89	515	380	366	366
query90	2123	182	184	182
query91	127	97	98	97
query92	58	46	46	46
query93	7284	503	504	503
query94	1353	184	181	181
query95	394	305	301	301
query96	592	274	266	266
query97	3147	2978	2939	2939
query98	236	224	214	214
query99	1223	879	864	864
Total cold run time: 314757 ms
Total hot run time: 185670 ms

doris-robot avatar May 02 '24 15:05 doris-robot

TeamCity be ut coverage result: Function Coverage: 35.65% (8961/25135) Line Coverage: 27.27% (73916/271098) Region Coverage: 26.46% (38189/144332) Branch Coverage: 23.20% (19443/83788) Coverage Report: http://coverage.selectdb-in.cc/coverage/8318a3f1939e51aa1301849e6628a624a0c099b1_8318a3f1939e51aa1301849e6628a624a0c099b1/report/index.html

doris-robot avatar May 02 '24 15:05 doris-robot

run buildall

Vallishp avatar May 02 '24 17:05 Vallishp

TeamCity be ut coverage result: Function Coverage: 35.65% (8960/25135) Line Coverage: 27.26% (73913/271098) Region Coverage: 26.46% (38186/144332) Branch Coverage: 23.21% (19446/83788) Coverage Report: http://coverage.selectdb-in.cc/coverage/48f15405b44dc0733209c1a1ca2436600fc7f18d_48f15405b44dc0733209c1a1ca2436600fc7f18d/report/index.html

doris-robot avatar May 02 '24 17:05 doris-robot

TPC-DS: Total hot run time: 188033 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48f15405b44dc0733209c1a1ca2436600fc7f18d, data reload: false

query1	915	353	341	341
query2	6141	2327	2289	2289
query3	6670	199	205	199
query4	23007	21849	21960	21849
query5	4130	438	408	408
query6	261	170	170	170
query7	4780	301	290	290
query8	261	181	206	181
query9	8626	2360	2332	2332
query10	428	246	261	246
query11	15319	14712	14878	14712
query12	130	88	84	84
query13	1643	390	375	375
query14	9920	7889	7886	7886
query15	287	180	192	180
query16	8451	272	272	272
query17	1982	605	596	596
query18	2115	292	287	287
query19	331	154	157	154
query20	92	89	93	89
query21	197	129	127	127
query22	5099	4849	4879	4849
query23	33901	33375	33214	33214
query24	10125	2889	2933	2889
query25	565	380	377	377
query26	692	153	149	149
query27	2060	315	319	315
query28	5761	2014	2056	2014
query29	843	608	599	599
query30	202	147	148	147
query31	934	711	721	711
query32	88	49	52	49
query33	629	245	238	238
query34	882	470	477	470
query35	818	645	653	645
query36	1063	877	899	877
query37	103	68	62	62
query38	3107	3040	2964	2964
query39	1574	1531	1528	1528
query40	192	124	125	124
query41	39	37	36	36
query42	105	95	92	92
query43	579	546	534	534
query44	1090	733	739	733
query45	258	248	246	246
query46	1066	680	716	680
query47	1912	1855	1827	1827
query48	359	300	296	296
query49	820	388	383	383
query50	758	379	381	379
query51	6851	6667	6583	6583
query52	96	96	84	84
query53	358	282	274	274
query54	313	230	244	230
query55	74	72	74	72
query56	233	211	214	211
query57	1197	1123	1135	1123
query58	230	203	199	199
query59	3388	3030	3138	3030
query60	245	231	241	231
query61	88	86	102	86
query62	629	454	453	453
query63	318	283	282	282
query64	8314	7370	7203	7203
query65	3109	3017	3025	3017
query66	763	334	332	332
query67	15512	14852	15190	14852
query68	9407	530	549	530
query69	559	307	305	305
query70	1176	1088	1171	1088
query71	520	276	265	265
query72	8002	2579	2401	2401
query73	1351	332	327	327
query74	7019	6365	6069	6069
query75	4480	2713	2590	2590
query76	5376	1001	965	965
query77	651	270	267	267
query78	11033	10290	10210	10210
query79	12839	511	519	511
query80	2259	432	417	417
query81	497	222	214	214
query82	721	90	89	89
query83	208	162	161	161
query84	260	83	82	82
query85	1296	269	271	269
query86	396	257	264	257
query87	3279	3117	3080	3080
query88	5018	2434	2417	2417
query89	521	379	379	379
query90	1974	184	182	182
query91	124	97	94	94
query92	59	46	50	46
query93	7254	512	503	503
query94	1202	182	184	182
query95	1098	1091	1089	1089
query96	617	272	269	269
query97	3133	2942	2968	2942
query98	236	218	225	218
query99	1282	867	859	859
Total cold run time: 306875 ms
Total hot run time: 188033 ms

doris-robot avatar May 02 '24 18:05 doris-robot

run buildall

Vallishp avatar May 03 '24 07:05 Vallishp

TeamCity be ut coverage result: Function Coverage: 35.66% (8962/25135) Line Coverage: 27.27% (73926/271098) Region Coverage: 26.46% (38192/144332) Branch Coverage: 23.21% (19444/83788) Coverage Report: http://coverage.selectdb-in.cc/coverage/48f15405b44dc0733209c1a1ca2436600fc7f18d_48f15405b44dc0733209c1a1ca2436600fc7f18d/report/index.html

doris-robot avatar May 03 '24 07:05 doris-robot

TPC-DS: Total hot run time: 187819 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48f15405b44dc0733209c1a1ca2436600fc7f18d, data reload: false

query1	912	361	338	338
query2	6300	2312	2451	2312
query3	6678	207	206	206
query4	23105	21729	21743	21729
query5	3856	447	403	403
query6	252	182	175	175
query7	4526	296	291	291
query8	242	188	178	178
query9	8569	2350	2392	2350
query10	406	235	263	235
query11	15399	14976	14943	14943
query12	124	89	93	89
query13	1820	381	400	381
query14	9668	8510	7580	7580
query15	266	170	163	163
query16	8115	264	259	259
query17	1903	568	564	564
query18	1866	288	285	285
query19	332	157	150	150
query20	92	83	91	83
query21	200	130	122	122
query22	5136	4931	4892	4892
query23	34744	34028	33694	33694
query24	12501	2894	2982	2894
query25	661	355	354	354
query26	1055	151	145	145
query27	2328	315	307	307
query28	6881	1984	1985	1984
query29	848	598	599	598
query30	250	153	148	148
query31	952	738	722	722
query32	87	48	50	48
query33	732	232	235	232
query34	1068	467	483	467
query35	786	668	644	644
query36	1062	945	900	900
query37	132	66	62	62
query38	3062	2989	2999	2989
query39	1593	1530	1534	1530
query40	198	121	121	121
query41	46	36	34	34
query42	108	96	91	91
query43	550	563	540	540
query44	1258	732	731	731
query45	278	252	254	252
query46	1071	722	743	722
query47	1969	1900	1880	1880
query48	371	293	301	293
query49	851	385	387	385
query50	791	389	383	383
query51	6807	6558	6692	6558
query52	98	86	88	86
query53	347	272	271	271
query54	304	225	236	225
query55	77	70	73	70
query56	237	216	217	216
query57	1220	1118	1127	1118
query58	225	199	217	199
query59	3487	3186	3106	3106
query60	248	227	229	227
query61	86	85	86	85
query62	656	446	437	437
query63	299	276	278	276
query64	8587	7268	7240	7240
query65	3065	3057	3038	3038
query66	817	329	338	329
query67	15641	15070	14927	14927
query68	9169	534	547	534
query69	551	302	305	302
query70	1203	1139	1131	1131
query71	534	264	274	264
query72	7919	2539	2326	2326
query73	857	320	323	320
query74	6666	5992	6037	5992
query75	5030	2657	2659	2657
query76	5466	985	1023	985
query77	804	259	267	259
query78	10853	10415	10280	10280
query79	11371	511	504	504
query80	1387	423	426	423
query81	495	222	221	221
query82	776	90	90	90
query83	204	161	163	161
query84	268	88	81	81
query85	1408	266	258	258
query86	401	291	269	269
query87	3265	3120	3096	3096
query88	4688	2428	2425	2425
query89	528	386	385	385
query90	1999	182	180	180
query91	125	98	95	95
query92	56	47	46	46
query93	7273	501	500	500
query94	1244	180	181	180
query95	402	309	300	300
query96	609	268	268	268
query97	3179	2976	2954	2954
query98	236	246	213	213
query99	1273	863	834	834
Total cold run time: 308440 ms
Total hot run time: 187819 ms

doris-robot avatar May 03 '24 08:05 doris-robot

run buildall

Vallishp avatar May 03 '24 09:05 Vallishp

TeamCity be ut coverage result: Function Coverage: 35.65% (8961/25135) Line Coverage: 27.27% (73925/271098) Region Coverage: 26.47% (38201/144332) Branch Coverage: 23.21% (19447/83788) Coverage Report: http://coverage.selectdb-in.cc/coverage/d24d0001b98b376cd20eabe99d3cc73f7011c5e2_d24d0001b98b376cd20eabe99d3cc73f7011c5e2/report/index.html

doris-robot avatar May 03 '24 10:05 doris-robot

TPC-H: Total hot run time: 41778 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d24d0001b98b376cd20eabe99d3cc73f7011c5e2, data reload: false

------ Round 1 ----------------------------------
q1	17602	4405	4241	4241
q2	2021	192	188	188
q3	10489	1167	1139	1139
q4	10193	851	771	771
q5	7510	2674	2720	2674
q6	223	133	135	133
q7	1065	648	620	620
q8	9239	2137	2121	2121
q9	9316	6750	6746	6746
q10	9265	3881	3887	3881
q11	444	249	250	249
q12	468	248	238	238
q13	17213	3119	3245	3119
q14	262	240	224	224
q15	532	472	466	466
q16	489	390	393	390
q17	990	680	748	680
q18	8142	7852	7731	7731
q19	2271	1570	1546	1546
q20	641	324	316	316
q21	5419	4288	4025	4025
q22	361	280	298	280
Total cold run time: 114155 ms
Total hot run time: 41778 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4554	4434	4381	4381
q2	367	273	276	273
q3	3184	2910	2878	2878
q4	1943	1573	1650	1573
q5	5471	5502	5495	5495
q6	220	124	129	124
q7	2327	1987	1991	1987
q8	3284	3395	3391	3391
q9	8745	8752	8801	8752
q10	3987	3672	3807	3672
q11	592	510	514	510
q12	799	637	622	622
q13	15876	3131	3179	3131
q14	300	270	267	267
q15	537	497	490	490
q16	499	453	440	440
q17	1787	1553	1519	1519
q18	7743	7650	7368	7368
q19	3277	1593	1527	1527
q20	2001	1769	1774	1769
q21	10398	4843	4847	4843
q22	557	491	504	491
Total cold run time: 78448 ms
Total hot run time: 55503 ms

doris-robot avatar May 03 '24 10:05 doris-robot

TPC-DS: Total hot run time: 185086 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d24d0001b98b376cd20eabe99d3cc73f7011c5e2, data reload: false

query1	912	365	350	350
query2	7357	2398	2338	2338
query3	6640	212	212	212
query4	23271	21179	21321	21179
query5	4207	416	410	410
query6	265	180	179	179
query7	4578	289	291	289
query8	237	196	200	196
query9	8504	2353	2283	2283
query10	441	256	250	250
query11	14661	14245	14123	14123
query12	139	93	88	88
query13	1646	378	375	375
query14	10613	6696	6961	6696
query15	238	170	162	162
query16	7977	260	256	256
query17	1742	560	533	533
query18	2084	271	264	264
query19	191	168	147	147
query20	94	88	91	88
query21	193	125	122	122
query22	5041	4830	4850	4830
query23	34238	33299	33350	33299
query24	6426	2991	2962	2962
query25	564	368	352	352
query26	695	149	153	149
query27	1989	310	304	304
query28	3874	2013	1985	1985
query29	843	601	600	600
query30	243	153	153	153
query31	965	741	728	728
query32	86	51	54	51
query33	474	254	256	254
query34	876	471	498	471
query35	781	672	700	672
query36	1050	887	874	874
query37	108	67	75	67
query38	3140	3012	3037	3012
query39	1581	1520	1546	1520
query40	195	125	127	125
query41	43	41	39	39
query42	102	95	94	94
query43	573	508	529	508
query44	1076	755	762	755
query45	275	253	252	252
query46	1050	702	720	702
query47	1931	1875	1841	1841
query48	381	305	292	292
query49	835	389	398	389
query50	759	399	405	399
query51	6784	6656	6565	6565
query52	104	91	93	91
query53	347	285	275	275
query54	256	240	234	234
query55	77	71	71	71
query56	236	219	219	219
query57	1177	1129	1132	1129
query58	223	203	198	198
query59	3457	3231	3173	3173
query60	248	240	245	240
query61	91	112	89	89
query62	605	448	463	448
query63	305	277	278	277
query64	8229	7251	7129	7129
query65	3098	3088	3063	3063
query66	791	345	336	336
query67	15479	15114	14857	14857
query68	5528	527	538	527
query69	528	299	300	299
query70	1202	1067	1146	1067
query71	418	272	273	272
query72	8110	2587	2348	2348
query73	703	324	322	322
query74	6407	6118	6066	6066
query75	3511	2673	2639	2639
query76	3724	1018	1008	1008
query77	587	262	264	262
query78	10989	10509	10091	10091
query79	4563	504	517	504
query80	1794	419	438	419
query81	539	226	217	217
query82	1544	102	104	102
query83	273	174	176	174
query84	266	87	87	87
query85	1290	273	272	272
query86	460	300	321	300
query87	3325	3131	3116	3116
query88	4799	2426	2416	2416
query89	485	381	371	371
query90	2059	187	188	187
query91	126	99	100	99
query92	62	50	48	48
query93	5659	514	493	493
query94	1231	188	180	180
query95	395	305	314	305
query96	606	265	265	265
query97	3152	2934	2941	2934
query98	249	220	221	220
query99	1229	863	903	863
Total cold run time: 283504 ms
Total hot run time: 185086 ms

doris-robot avatar May 03 '24 10:05 doris-robot

What is the meaning of TABLE_SCHEMA?

dataroaring avatar May 06 '24 13:05 dataroaring

great job. BTW, should we put properties in json format?

dataroaring avatar May 06 '24 13:05 dataroaring

DISTRIBUTE_KEY, TABLE_MODEL(agg unique, dup) should also in the table.

dataroaring avatar May 06 '24 14:05 dataroaring

Actually, we should show all messages in create table.

dataroaring avatar May 06 '24 14:05 dataroaring

Please add the document in doris-website and link 2 prs.

dataroaring avatar May 06 '24 14:05 dataroaring

What is the meaning of TABLE_SCHEMA?

databases to which table belongs.

Vallishp avatar May 06 '24 14:05 Vallishp

DISTRIBUTE_KEY, TABLE_MODEL(agg unique, dup)

DISTRIBUTE_KEY already have added. TABLE_MODEL(agg unique, dup) i will add this.

Actually, we should show all messages in create table. // currently table_options working for OlapTable. should i consider for broker and mysql type also?

Vallishp avatar May 06 '24 14:05 Vallishp

great job. BTW, should we put properties in json format?

Ok. can you please give me example for any column we print in json format as reference? thanks

Vallishp avatar May 06 '24 15:05 Vallishp

great job. BTW, should we put properties in json format?

Ok. can you please give me example for any column we print in json format as reference? thanks

properties.

`PROPERTIES: min_load_replica_num = -1, data_sort.col_num = 6, group_commit_interval_ms = 10000, data_sort.sort_type = LEXICAL, is_being_synced = false, binlog.enable = false, binlog.ttl_seconds = 86400, inverted_index_storage_format = V1, time_series_compaction_empty_rowsets_threshold = 5, default.replication_allocation = tag.location.default: 1, time_series_compaction_level_threshold = 1, time_series_compaction_time_threshold_seconds = 3600, storage_format = V2, store_row_column = false, light_schema_change = true, enable_unique_key_merge_on_write = false, in_memory = false, file_cache_ttl_seconds = 0, group_commit_data_bytes = 134217728, compaction_policy = size_based, _auto_bucket = false, binlog.max_history_nums = 9223372036854775807, time_series_compaction_file_count_threshold = 2000, skip_write_index_on_load = false, disable_auto_compaction = false, time_series_compaction_goal_size_mbytes = 1024, storage_medium = HDD, enable_single_replica_compaction = false, compression = LZ4F, binlog.max_bytes = 9223372036854775807

{ "min_load_replica_num": -1,
 "data_sort.col_num": 6,
 ...
}

dataroaring avatar May 07 '24 02:05 dataroaring

run buildall

Vallishp avatar May 07 '24 17:05 Vallishp

run buildall

Vallishp avatar May 15 '24 12:05 Vallishp

TPC-H: Total hot run time: 40387 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49a16afb709afc1d081c6e43e8343a02e016d0df, data reload: false

------ Round 1 ----------------------------------
q1	17630	4501	4276	4276
q2	2019	187	187	187
q3	10958	1260	1187	1187
q4	10562	905	811	811
q5	7978	2735	2725	2725
q6	216	135	134	134
q7	1006	603	597	597
q8	9287	2162	2143	2143
q9	9449	6722	6591	6591
q10	9035	3758	3706	3706
q11	462	239	241	239
q12	459	220	220	220
q13	17775	2998	2988	2988
q14	283	206	216	206
q15	528	475	463	463
q16	527	383	377	377
q17	967	709	768	709
q18	8148	7529	7497	7497
q19	1830	1561	1473	1473
q20	652	300	313	300
q21	5037	3285	3927	3285
q22	336	273	277	273
Total cold run time: 115144 ms
Total hot run time: 40387 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4304	4263	4291	4263
q2	364	260	260	260
q3	2949	2747	2712	2712
q4	1902	1643	1588	1588
q5	5267	5305	5257	5257
q6	208	124	124	124
q7	2242	1833	1877	1833
q8	3208	3345	3343	3343
q9	8376	8409	8390	8390
q10	3881	3698	3661	3661
q11	592	477	489	477
q12	794	572	586	572
q13	16432	3028	2968	2968
q14	307	253	248	248
q15	512	459	478	459
q16	473	416	411	411
q17	1795	1474	1482	1474
q18	7571	7693	7542	7542
q19	1681	1523	1527	1523
q20	1954	1769	1758	1758
q21	4797	4788	4747	4747
q22	585	496	490	490
Total cold run time: 70194 ms
Total hot run time: 54100 ms

doris-robot avatar May 15 '24 13:05 doris-robot

TPC-DS: Total hot run time: 187821 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49a16afb709afc1d081c6e43e8343a02e016d0df, data reload: false

query1	907	390	364	364
query2	6459	2544	2333	2333
query3	6648	214	215	214
query4	26201	21244	21446	21244
query5	4195	421	410	410
query6	265	167	177	167
query7	4588	306	287	287
query8	239	189	187	187
query9	8573	2390	2361	2361
query10	439	246	256	246
query11	14733	14167	14134	14134
query12	133	90	85	85
query13	1645	381	365	365
query14	9974	8518	8384	8384
query15	261	162	167	162
query16	8085	259	251	251
query17	1865	546	554	546
query18	2068	269	267	267
query19	243	145	146	145
query20	89	85	86	85
query21	200	127	125	125
query22	5052	4885	4794	4794
query23	34034	33564	33580	33564
query24	12122	2943	2885	2885
query25	641	359	364	359
query26	1742	154	148	148
query27	3000	322	335	322
query28	7210	2054	2032	2032
query29	990	627	606	606
query30	312	170	176	170
query31	962	770	723	723
query32	95	56	53	53
query33	751	250	258	250
query34	1049	489	485	485
query35	799	677	668	668
query36	1070	904	907	904
query37	264	70	74	70
query38	2879	2734	2731	2731
query39	1609	1565	1542	1542
query40	280	125	122	122
query41	45	45	44	44
query42	104	96	95	95
query43	575	562	557	557
query44	1176	729	761	729
query45	271	268	269	268
query46	1075	729	739	729
query47	1978	1887	1942	1887
query48	366	298	296	296
query49	1189	392	382	382
query50	765	382	392	382
query51	6870	6730	6773	6730
query52	99	94	90	90
query53	357	293	284	284
query54	936	431	425	425
query55	72	74	74	74
query56	237	219	224	219
query57	1246	1176	1143	1143
query58	226	205	194	194
query59	3590	3274	3238	3238
query60	265	233	234	233
query61	89	107	103	103
query62	656	483	460	460
query63	308	282	288	282
query64	9796	7413	7347	7347
query65	3137	3048	3096	3048
query66	1388	341	348	341
query67	15434	15336	15045	15045
query68	4548	531	532	531
query69	467	337	298	298
query70	1182	1091	1152	1091
query71	403	270	265	265
query72	7303	2549	2369	2369
query73	704	323	317	317
query74	6491	6144	6150	6144
query75	3491	2665	2646	2646
query76	2831	972	937	937
query77	419	264	265	264
query78	10551	10287	10035	10035
query79	2604	513	525	513
query80	826	437	429	429
query81	508	241	245	241
query82	923	97	92	92
query83	261	165	167	165
query84	237	86	93	86
query85	1326	362	256	256
query86	422	310	313	310
query87	3249	3065	3064	3064
query88	4093	2327	2348	2327
query89	468	376	373	373
query90	2017	183	180	180
query91	122	111	96	96
query92	63	50	49	49
query93	1603	501	488	488
query94	1194	183	184	183
query95	395	297	304	297
query96	587	270	267	267
query97	3135	2973	2996	2973
query98	244	227	209	209
query99	1143	911	912	911
Total cold run time: 288731 ms
Total hot run time: 187821 ms

doris-robot avatar May 15 '24 13:05 doris-robot

ClickBench: Total hot run time: 30.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49a16afb709afc1d081c6e43e8343a02e016d0df, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.49	0.47	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.50	0.47
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.59	0.59	0.60
query14	0.76	0.79	0.78
query15	0.81	0.80	0.80
query16	0.37	0.36	0.37
query17	0.97	0.97	0.94
query18	0.20	0.24	0.25
query19	1.86	1.65	1.74
query20	0.02	0.01	0.01
query21	15.52	0.68	0.66
query22	3.97	7.09	2.27
query23	18.29	1.37	1.14
query24	1.81	0.23	0.21
query25	0.14	0.08	0.07
query26	0.27	0.17	0.17
query27	0.07	0.08	0.07
query28	13.34	1.02	1.00
query29	13.19	3.26	3.28
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.28	0.47	0.47
query33	2.86	2.84	2.79
query34	17.10	4.46	4.42
query35	4.53	4.53	4.47
query36	0.64	0.47	0.49
query37	0.17	0.17	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.52 s
Total hot run time: 30.53 s

doris-robot avatar May 15 '24 13:05 doris-robot

TeamCity be ut coverage result: Function Coverage: 35.70% (8997/25204) Line Coverage: 27.37% (74455/272023) Region Coverage: 26.60% (38484/144654) Branch Coverage: 23.42% (19636/83828) Coverage Report: http://coverage.selectdb-in.cc/coverage/49a16afb709afc1d081c6e43e8343a02e016d0df_49a16afb709afc1d081c6e43e8343a02e016d0df/report/index.html

doris-robot avatar May 15 '24 14:05 doris-robot

Please add the document in doris-website and link 2 prs.

https://github.com/apache/doris-website/pull/661

handled all, pls help to review again

Vallishp avatar May 16 '24 03:05 Vallishp