doris [opt](catalog) support using loading cache for db/table list in external catalog

Proposed changes

Previous Implementation Logic in Doris with External Catalog

Steps

Master FE uniformly retrieves table information and generates the corresponding id -> name mapping.
The id -> name mapping is stored in Doris's metadata and persisted.
Master FE synchronizes this information with other FEs via EditLog.
To update the table information, a refresh command must be executed or the metadata synchronized through an HMS event.

Advantages and Disadvantages

Advantage: All FEs can see a consistent list of tables as the information is uniformly obtained from the Master FE, preventing discrepancies in table visibility across different FEs.
Disadvantage: There is an inability to promptly perceive changes in the tables. For example, a new table on the Hive side is not immediately visible on the Doris side and requires a refresh or periodic metadata refresh for visibility.

Modifications in this PR

1. User Interface

Catalog adds a new property use_meta_cache. Default is false. If set to true, it will use an independent caching method to synchronize table information.

2. Enabling `use_meta_cache`

Once enabled, table information will no longer be uniformly obtained by Master FE but will instead be independently fetched by each FE.
Each FE has its own cache of the Database and Table list, implemented using the Caffeine library.
This cache synchronously loads table information when accessed. If the cache does not exist, it will directly access HMS for table information.
Behaviors:
- Different FEs may see different table information due to different loading times, but they will eventually be consistent.
- New tables created on the Hive side can be queried directly in Doris, but may not be visible in show databases or show tables.
- Tables deleted on the Hive side will still appear in show databases/tables but will be inaccessible.
- All caches will refresh at most every 10 minutes.
Compatibility:
- For already created catalog, after upgrade, the use_meta_cache is false.
- For newly created catalog, if use_meta_cache is not set, set it as false.
- Can not modify use_meta_cache after being created.

3. Code Implementation

MetaCache:
- A general Cache class responsible for caching Database/Table information, including two LoadingCaches for storing "name lists" and "name-to-object" caches.
ID Generation Rules:
- As table information is no longer uniformly fetched by the Master FE, a consistent rule must exist to ensure that each FE generates the same ID for the same table. Here, we use the absolute value of the top 8 bits of the sha256 hash of the table name as the object's ID. This way, the same name generates the same ID, but the ID is no longer globally unique and is unique only within the Catalog or Database level.
Remove some unused methods such as getIdToTable() and getIdToDb()

Further comments

I have run the p0 for both use_meta_cache = true and use_meta_cache=false

Apr 12 '24 16:04 morningman

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

Apr 12 '24 16:04 doris-robot

run buildall

Apr 12 '24 16:04 morningman

TPC-H: Total hot run time: 38961 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83f8026c2d2559c9331e0cd05478598ee10aab42, data reload: false

------ Round 1 ----------------------------------
q1	17616	4375	4277	4277
q2	2009	198	207	198
q3	10426	1184	1192	1184
q4	10196	755	765	755
q5	7553	2691	2672	2672
q6	215	132	129	129
q7	1020	616	605	605
q8	9217	2039	2027	2027
q9	7932	6640	6543	6543
q10	8601	3562	3525	3525
q11	467	255	241	241
q12	488	237	233	233
q13	18797	2995	2992	2992
q14	281	232	240	232
q15	508	485	494	485
q16	537	393	389	389
q17	963	704	740	704
q18	7490	6763	6740	6740
q19	7338	1562	1544	1544
q20	687	338	308	308
q21	3647	2845	2862	2845
q22	385	336	333	333
Total cold run time: 116373 ms
Total hot run time: 38961 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4225	4284	4225
q2	384	275	276	275
q3	3000	2786	2737	2737
q4	1893	1602	1622	1602
q5	5358	5398	5337	5337
q6	208	122	121	121
q7	2299	1871	1913	1871
q8	3216	3372	3324	3324
q9	8537	8558	8743	8558
q10	4124	4033	3967	3967
q11	632	527	513	513
q12	805	641	717	641
q13	16462	3207	3205	3205
q14	328	325	293	293
q15	526	478	483	478
q16	528	475	479	475
q17	1872	1530	1510	1510
q18	8245	8043	7741	7741
q19	1686	1586	1603	1586
q20	2092	1869	1891	1869
q21	5111	5061	5122	5061
q22	558	507	482	482
Total cold run time: 72269 ms
Total hot run time: 55871 ms

Apr 12 '24 17:04 doris-robot

TPC-DS: Total hot run time: 188477 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83f8026c2d2559c9331e0cd05478598ee10aab42, data reload: false

query1	880	1127	1139	1127
query2	6179	2904	2745	2745
query3	6661	225	218	218
query4	37133	21711	21496	21496
query5	4146	413	410	410
query6	247	201	190	190
query7	4027	316	306	306
query8	221	179	176	176
query9	5953	2552	2525	2525
query10	382	251	262	251
query11	14595	14390	14287	14287
query12	139	93	91	91
query13	991	390	386	386
query14	9310	7019	7113	7019
query15	222	182	186	182
query16	6790	284	275	275
query17	1633	614	583	583
query18	1399	305	283	283
query19	205	167	162	162
query20	99	95	99	95
query21	206	128	132	128
query22	5054	4926	4892	4892
query23	33692	33377	33619	33377
query24	11139	3121	3092	3092
query25	584	424	425	424
query26	897	172	168	168
query27	2956	407	394	394
query28	6547	2375	2323	2323
query29	900	650	646	646
query30	297	181	188	181
query31	970	760	778	760
query32	62	68	59	59
query33	513	285	284	284
query34	935	523	532	523
query35	945	743	754	743
query36	1123	995	1006	995
query37	114	81	73	73
query38	3630	3679	3705	3679
query39	1640	1586	1584	1584
query40	181	136	139	136
query41	55	51	55	51
query42	113	103	114	103
query43	642	623	611	611
query44	1219	795	799	795
query45	313	280	307	280
query46	1193	812	802	802
query47	2088	1963	1984	1963
query48	407	333	337	333
query49	864	391	408	391
query50	787	416	416	416
query51	7035	6997	6810	6810
query52	106	97	92	92
query53	357	290	286	286
query54	262	245	234	234
query55	86	75	77	75
query56	246	249	235	235
query57	1187	1134	1161	1134
query58	229	215	211	211
query59	3694	3334	3440	3334
query60	266	249	252	249
query61	97	94	109	94
query62	598	462	440	440
query63	316	286	289	286
query64	4212	4125	4019	4019
query65	3116	3017	3054	3017
query66	743	325	321	321
query67	15753	15470	15166	15166
query68	9141	591	603	591
query69	626	325	323	323
query70	1284	1200	1213	1200
query71	513	287	284	284
query72	6718	2644	2504	2504
query73	945	343	351	343
query74	7350	6526	6453	6453
query75	3485	2435	2388	2388
query76	5951	1159	1149	1149
query77	637	267	263	263
query78	11052	10143	10199	10143
query79	11377	552	561	552
query80	1940	457	429	429
query81	570	236	244	236
query82	795	108	103	103
query83	220	178	174	174
query84	268	83	87	83
query85	984	322	264	264
query86	479	348	321	321
query87	3833	3516	3611	3516
query88	6854	2595	2520	2520
query89	552	375	384	375
query90	1952	189	186	186
query91	129	100	103	100
query92	67	49	51	49
query93	7105	560	566	560
query94	1163	194	186	186
query95	397	316	319	316
query96	657	288	285	285
query97	2724	2493	2477	2477
query98	238	225	222	222
query99	1192	895	866	866
Total cold run time: 310779 ms
Total hot run time: 188477 ms

Apr 12 '24 17:04 doris-robot

ClickBench: Total hot run time: 30.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83f8026c2d2559c9331e0cd05478598ee10aab42, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.51	0.50	0.50
query6	1.14	0.67	0.66
query7	0.02	0.02	0.02
query8	0.06	0.04	0.04
query9	0.57	0.51	0.50
query10	0.57	0.58	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.59	0.59	0.60
query14	0.77	0.78	0.80
query15	0.84	0.82	0.82
query16	0.37	0.35	0.37
query17	1.00	1.01	0.96
query18	0.22	0.25	0.23
query19	1.77	1.73	1.73
query20	0.01	0.01	0.01
query21	15.46	0.68	0.68
query22	4.37	7.75	1.67
query23	18.33	1.39	1.30
query24	1.97	0.22	0.22
query25	0.14	0.08	0.09
query26	0.29	0.16	0.17
query27	0.08	0.09	0.08
query28	13.34	1.02	1.01
query29	12.64	3.40	3.36
query30	0.26	0.08	0.06
query31	2.83	0.40	0.40
query32	3.26	0.48	0.47
query33	2.85	2.85	2.87
query34	17.00	4.40	4.45
query35	4.53	4.46	4.51
query36	0.66	0.47	0.47
query37	0.19	0.16	0.15
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.18	0.16	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.57 s
Total hot run time: 30.46 s

Apr 12 '24 17:04 doris-robot

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 83f8026c2d2559c9331e0cd05478598ee10aab42 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      34 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

Apr 12 '24 17:04 doris-robot

run buildall

Apr 13 '24 01:04 morningman

TPC-H: Total hot run time: 38357 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 813415ee86186966ed604504e8e86dff5c3d4ad7, data reload: false

------ Round 1 ----------------------------------
q1	17644	4355	4232	4232
q2	2011	195	192	192
q3	10449	1135	1228	1135
q4	10191	842	785	785
q5	7538	2696	2594	2594
q6	217	133	133	133
q7	1032	623	586	586
q8	9237	2063	2029	2029
q9	7951	6581	6532	6532
q10	8632	3534	3530	3530
q11	458	234	233	233
q12	417	234	228	228
q13	17796	2972	2989	2972
q14	275	223	236	223
q15	511	479	477	477
q16	517	409	390	390
q17	946	622	692	622
q18	7274	6725	6610	6610
q19	7614	1521	1516	1516
q20	685	318	315	315
q21	3404	2709	2841	2709
q22	380	316	314	314
Total cold run time: 115179 ms
Total hot run time: 38357 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4376	4202	4202
q2	387	264	267	264
q3	2995	2792	2799	2792
q4	1850	1619	1613	1613
q5	5326	5386	5280	5280
q6	210	123	123	123
q7	2256	1913	1880	1880
q8	3210	3341	3276	3276
q9	8612	8677	8783	8677
q10	4075	3931	4013	3931
q11	607	510	507	507
q12	810	636	662	636
q13	15992	3296	3236	3236
q14	336	279	279	279
q15	523	487	488	487
q16	516	452	467	452
q17	1822	1570	1516	1516
q18	8061	8005	7843	7843
q19	1708	1598	1518	1518
q20	1968	1875	1893	1875
q21	9296	5029	4989	4989
q22	595	489	507	489
Total cold run time: 75519 ms
Total hot run time: 55865 ms

Apr 13 '24 01:04 doris-robot

TPC-DS: Total hot run time: 186087 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 813415ee86186966ed604504e8e86dff5c3d4ad7, data reload: false

query1	881	371	1123	371
query2	6193	2982	2694	2694
query3	6662	214	209	209
query4	36982	21552	21535	21535
query5	4116	404	401	401
query6	256	197	184	184
query7	4059	310	301	301
query8	234	186	179	179
query9	5858	2567	2548	2548
query10	379	254	269	254
query11	14841	14218	14245	14218
query12	142	98	89	89
query13	1009	405	376	376
query14	9902	6928	6849	6849
query15	215	187	184	184
query16	6956	261	259	259
query17	1708	579	568	568
query18	1673	280	281	280
query19	195	154	154	154
query20	99	88	91	88
query21	191	122	124	122
query22	5084	4899	4889	4889
query23	33938	33117	33565	33117
query24	11411	3082	3051	3051
query25	525	387	382	382
query26	757	165	161	161
query27	2949	380	380	380
query28	6530	2288	2241	2241
query29	864	650	637	637
query30	256	183	171	171
query31	949	738	738	738
query32	63	53	57	53
query33	523	254	256	254
query34	915	498	538	498
query35	852	762	721	721
query36	1088	976	994	976
query37	110	72	72	72
query38	3671	3583	3642	3583
query39	1644	1714	1572	1572
query40	177	131	129	129
query41	49	45	47	45
query42	110	96	102	96
query43	618	611	595	595
query44	1303	763	765	763
query45	303	282	271	271
query46	1123	778	768	768
query47	2072	1937	2017	1937
query48	407	318	325	318
query49	867	393	377	377
query50	807	401	414	401
query51	6984	6866	6826	6826
query52	96	98	87	87
query53	348	283	282	282
query54	254	227	229	227
query55	79	74	72	72
query56	236	224	221	221
query57	1202	1107	1120	1107
query58	224	202	210	202
query59	3623	3381	3402	3381
query60	253	231	233	231
query61	93	93	105	93
query62	609	436	452	436
query63	309	279	280	279
query64	4030	3781	3922	3781
query65	3075	3047	3057	3047
query66	758	314	328	314
query67	15888	15186	15357	15186
query68	4511	576	583	576
query69	500	315	318	315
query70	1308	1265	1285	1265
query71	355	279	278	278
query72	6381	2666	2577	2577
query73	735	335	343	335
query74	7010	6446	6363	6363
query75	3031	2352	2390	2352
query76	2228	1138	1141	1138
query77	637	263	274	263
query78	10850	10223	10022	10022
query79	5655	541	540	540
query80	2039	472	461	461
query81	549	246	248	246
query82	1582	111	103	103
query83	340	175	174	174
query84	264	95	87	87
query85	1143	317	312	312
query86	457	332	307	307
query87	3816	3580	3537	3537
query88	6178	2444	2457	2444
query89	491	397	379	379
query90	1828	182	180	180
query91	134	109	110	109
query92	57	48	49	48
query93	5463	544	534	534
query94	1118	189	189	189
query95	387	303	301	301
query96	606	283	274	274
query97	2679	2489	2470	2470
query98	240	228	223	223
query99	1191	864	854	854
Total cold run time: 294340 ms
Total hot run time: 186087 ms

Apr 13 '24 01:04 doris-robot

ClickBench: Total hot run time: 31.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 813415ee86186966ed604504e8e86dff5c3d4ad7, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.09	0.07
query5	0.50	0.49	0.51
query6	1.12	0.66	0.66
query7	0.03	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.50	0.50
query10	0.55	0.57	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.61	0.58	0.58
query14	0.78	0.78	0.80
query15	0.83	0.80	0.80
query16	0.37	0.37	0.38
query17	1.03	0.99	1.03
query18	0.24	0.23	0.24
query19	1.83	1.76	1.68
query20	0.01	0.01	0.01
query21	15.42	0.68	0.67
query22	4.05	6.82	2.73
query23	18.25	1.35	1.33
query24	1.72	0.23	0.21
query25	0.14	0.08	0.07
query26	0.27	0.17	0.16
query27	0.09	0.07	0.08
query28	13.55	1.02	0.98
query29	12.64	3.30	3.28
query30	0.26	0.06	0.08
query31	2.87	0.38	0.38
query32	3.26	0.48	0.47
query33	2.84	2.84	2.85
query34	17.35	4.37	4.39
query35	4.49	4.50	4.44
query36	0.65	0.46	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.15	0.14
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 31.26 s

Apr 13 '24 02:04 doris-robot

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 813415ee86186966ed604504e8e86dff5c3d4ad7 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      34 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

Apr 13 '24 02:04 doris-robot

run buildall

Apr 13 '24 15:04 morningman

TPC-H: Total hot run time: 39540 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78292e6f96c2e6a49b53c55778616134741dfd84, data reload: false

------ Round 1 ----------------------------------
q1	18270	4369	4384	4369
q2	2008	198	186	186
q3	10466	1145	1161	1145
q4	10463	818	809	809
q5	8797	2731	2671	2671
q6	218	135	136	135
q7	1050	618	598	598
q8	9221	2097	2053	2053
q9	8036	6668	6621	6621
q10	8795	3703	3673	3673
q11	508	241	247	241
q12	522	239	240	239
q13	18038	3151	3211	3151
q14	273	245	257	245
q15	521	502	518	502
q16	808	408	416	408
q17	1123	633	678	633
q18	8445	6942	6895	6895
q19	7431	1548	1524	1524
q20	691	320	314	314
q21	3626	2820	2812	2812
q22	369	317	316	316
Total cold run time: 119679 ms
Total hot run time: 39540 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4250	4248	4248
q2	379	274	276	274
q3	2951	2825	2783	2783
q4	1863	1580	1554	1554
q5	5303	5351	5319	5319
q6	211	127	126	126
q7	2238	1860	1867	1860
q8	3192	3312	3301	3301
q9	8560	8602	8928	8602
q10	4104	4006	3980	3980
q11	617	512	498	498
q12	880	639	655	639
q13	16327	3232	3107	3107
q14	334	286	290	286
q15	516	488	474	474
q16	493	471	460	460
q17	1803	1507	1522	1507
q18	8105	7878	7601	7601
q19	1688	1591	1626	1591
q20	2088	1902	1835	1835
q21	5904	5198	5001	5001
q22	556	493	469	469
Total cold run time: 72449 ms
Total hot run time: 55515 ms

Apr 13 '24 16:04 doris-robot

TPC-DS: Total hot run time: 183980 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78292e6f96c2e6a49b53c55778616134741dfd84, data reload: false

query1	887	1131	1111	1111
query2	7698	2437	2273	2273
query3	6657	206	200	200
query4	37136	21533	21394	21394
query5	4152	404	406	404
query6	236	175	177	175
query7	4045	295	291	291
query8	219	173	172	172
query9	5736	2312	2302	2302
query10	373	232	250	232
query11	14587	14276	14277	14276
query12	138	98	91	91
query13	983	381	368	368
query14	9963	6950	6754	6754
query15	209	173	178	173
query16	7036	270	261	261
query17	1771	561	545	545
query18	1693	274	274	274
query19	189	151	153	151
query20	97	91	90	90
query21	202	131	128	128
query22	4995	4845	4812	4812
query23	34193	33006	33462	33006
query24	11269	3083	2977	2977
query25	529	399	394	394
query26	726	165	160	160
query27	2901	376	414	376
query28	6395	2095	2071	2071
query29	845	639	619	619
query30	221	170	168	168
query31	939	775	750	750
query32	62	53	53	53
query33	506	247	253	247
query34	901	496	549	496
query35	854	701	734	701
query36	1012	947	921	921
query37	116	70	73	70
query38	3733	3573	3619	3573
query39	1662	1569	1583	1569
query40	172	131	133	131
query41	48	46	45	45
query42	126	99	104	99
query43	600	524	556	524
query44	1278	737	729	729
query45	300	278	281	278
query46	1123	758	745	745
query47	2069	1920	1945	1920
query48	404	305	325	305
query49	801	382	380	380
query50	788	403	399	399
query51	6795	6783	6584	6584
query52	100	91	87	87
query53	354	282	285	282
query54	256	232	245	232
query55	80	71	70	70
query56	244	223	220	220
query57	1177	1112	1175	1112
query58	222	206	214	206
query59	3371	3048	3026	3026
query60	265	242	245	242
query61	103	92	89	89
query62	613	451	451	451
query63	309	281	282	281
query64	4349	3962	4047	3962
query65	3099	3094	3046	3046
query66	777	328	325	325
query67	15675	15275	14897	14897
query68	6914	564	550	550
query69	533	311	315	311
query70	1286	1201	1156	1156
query71	440	277	277	277
query72	6809	2625	2423	2423
query73	848	317	315	315
query74	6901	6405	6461	6405
query75	3133	2339	2387	2339
query76	4172	1142	1073	1073
query77	629	253	248	248
query78	11031	10223	10241	10223
query79	4460	535	540	535
query80	2214	449	441	441
query81	552	241	232	232
query82	1554	106	95	95
query83	348	182	174	174
query84	270	89	85	85
query85	1430	277	267	267
query86	451	299	294	294
query87	3737	3538	3537	3537
query88	5939	2272	2278	2272
query89	475	374	380	374
query90	1819	184	180	180
query91	125	100	150	100
query92	59	50	47	47
query93	5458	514	502	502
query94	1153	187	176	176
query95	387	1224	292	292
query96	622	280	261	261
query97	2634	2440	2461	2440
query98	239	223	215	215
query99	1263	846	846	846
Total cold run time: 299249 ms
Total hot run time: 183980 ms

Apr 13 '24 16:04 doris-robot

ClickBench: Total hot run time: 30.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78292e6f96c2e6a49b53c55778616134741dfd84, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.08	0.08
query5	0.48	0.49	0.49
query6	1.46	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.59	0.58
query14	0.77	0.78	0.79
query15	0.84	0.82	0.82
query16	0.37	0.35	0.40
query17	0.95	1.01	1.02
query18	0.23	0.23	0.22
query19	1.89	1.74	1.67
query20	0.02	0.01	0.01
query21	15.40	0.64	0.64
query22	4.21	7.63	1.81
query23	18.27	1.38	1.31
query24	1.65	0.25	0.20
query25	0.13	0.09	0.08
query26	0.26	0.16	0.15
query27	0.08	0.08	0.07
query28	13.57	1.02	1.01
query29	12.53	3.34	3.31
query30	0.25	0.07	0.06
query31	2.84	0.38	0.38
query32	3.29	0.47	0.46
query33	2.82	2.86	2.86
query34	17.21	4.50	4.43
query35	4.51	4.52	4.52
query36	0.65	0.47	0.46
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.17	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.49 s
Total hot run time: 30.44 s

Apr 13 '24 16:04 doris-robot

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 78292e6f96c2e6a49b53c55778616134741dfd84 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

Apr 13 '24 16:04 doris-robot

run buildall

Apr 17 '24 11:04 morningman

run buildall

Apr 18 '24 12:04 morningman

run buildall

Apr 19 '24 12:04 morningman

TPC-H: Total hot run time: 38547 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4cd8438fdb6556c74f6fd8d18fc654d508dbe009, data reload: false

------ Round 1 ----------------------------------
q1	17615	4395	4230	4230
q2	2020	206	185	185
q3	10426	1189	1218	1189
q4	10190	764	755	755
q5	7504	2679	2619	2619
q6	221	134	142	134
q7	1030	617	589	589
q8	9222	2080	2028	2028
q9	7201	6564	6509	6509
q10	8562	3532	3542	3532
q11	464	244	248	244
q12	461	227	222	222
q13	17808	2913	2919	2913
q14	261	227	238	227
q15	539	479	495	479
q16	528	400	381	381
q17	954	736	711	711
q18	7315	6795	6707	6707
q19	6510	1502	1540	1502
q20	647	326	314	314
q21	3548	2773	2875	2773
q22	359	304	315	304
Total cold run time: 113385 ms
Total hot run time: 38547 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4259	4209	4209
q2	363	277	264	264
q3	3000	2762	2701	2701
q4	1857	1586	1564	1564
q5	5334	5312	5295	5295
q6	206	124	125	124
q7	2292	1837	1890	1837
q8	3217	3366	3347	3347
q9	8607	8558	8700	8558
q10	4080	3948	4051	3948
q11	598	498	488	488
q12	795	636	649	636
q13	17025	3219	3155	3155
q14	335	286	291	286
q15	525	499	498	498
q16	520	432	448	432
q17	1839	1541	1523	1523
q18	8268	7808	7924	7808
q19	1669	1566	1565	1565
q20	2019	1861	1832	1832
q21	8172	4963	4938	4938
q22	550	476	477	476
Total cold run time: 75620 ms
Total hot run time: 55484 ms

Apr 19 '24 12:04 doris-robot

TPC-DS: Total hot run time: 185592 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4cd8438fdb6556c74f6fd8d18fc654d508dbe009, data reload: false

query1	891	368	362	362
query2	6230	2628	2402	2402
query3	6644	207	202	202
query4	24034	21392	21359	21359
query5	4160	398	406	398
query6	262	186	177	177
query7	4587	286	285	285
query8	251	192	190	190
query9	8801	2334	2300	2300
query10	413	246	254	246
query11	14644	14112	14134	14112
query12	133	89	84	84
query13	1641	367	365	365
query14	9354	7875	7954	7875
query15	252	175	191	175
query16	8193	257	255	255
query17	1934	541	566	541
query18	2111	270	266	266
query19	325	149	146	146
query20	91	87	86	86
query21	195	124	121	121
query22	5076	4864	4819	4819
query23	33901	33067	33322	33067
query24	10196	3067	3014	3014
query25	607	394	378	378
query26	701	155	151	151
query27	2306	364	385	364
query28	5909	2043	2054	2043
query29	856	631	608	608
query30	279	178	188	178
query31	970	751	757	751
query32	99	55	55	55
query33	685	256	254	254
query34	870	480	491	480
query35	819	702	731	702
query36	1086	932	919	919
query37	115	74	81	74
query38	3507	3363	3340	3340
query39	1625	1624	1578	1578
query40	168	130	128	128
query41	46	43	45	43
query42	100	100	96	96
query43	592	528	523	523
query44	1098	753	750	750
query45	270	270	260	260
query46	1088	712	743	712
query47	2023	1930	1904	1904
query48	368	303	307	303
query49	826	389	411	389
query50	789	398	402	398
query51	7022	6926	6797	6797
query52	102	83	90	83
query53	343	270	275	270
query54	298	239	223	223
query55	76	74	71	71
query56	250	226	229	226
query57	1203	1148	1145	1145
query58	214	194	214	194
query59	3493	3159	3148	3148
query60	254	223	224	223
query61	91	87	86	86
query62	563	454	432	432
query63	303	275	278	275
query64	4586	4043	3903	3903
query65	3072	3043	3058	3043
query66	743	323	315	315
query67	15326	15123	14920	14920
query68	6814	536	532	532
query69	546	304	294	294
query70	1191	1189	1124	1124
query71	1459	1264	1262	1262
query72	6748	2600	2445	2445
query73	731	315	314	314
query74	6842	6558	6408	6408
query75	3870	2642	2606	2606
query76	4294	1024	1013	1013
query77	606	255	263	255
query78	10932	10330	10258	10258
query79	7712	520	517	517
query80	924	439	427	427
query81	518	242	245	242
query82	243	90	98	90
query83	191	162	163	162
query84	263	82	83	82
query85	829	268	269	268
query86	346	300	288	288
query87	3449	3271	3276	3271
query88	4742	2313	2310	2310
query89	509	364	367	364
query90	2031	184	180	180
query91	122	93	106	93
query92	55	47	46	46
query93	5995	505	497	497
query94	1206	173	171	171
query95	379	288	299	288
query96	605	260	263	260
query97	3123	2936	2903	2903
query98	229	256	215	215
query99	1023	878	825	825
Total cold run time: 288587 ms
Total hot run time: 185592 ms

Apr 19 '24 13:04 doris-robot

ClickBench: Total hot run time: 30.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4cd8438fdb6556c74f6fd8d18fc654d508dbe009, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.03
query3	0.23	0.04	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.48
query6	1.48	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.53	0.48	0.49
query10	0.55	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	0.76	0.79	0.76
query15	0.82	0.80	0.80
query16	0.36	0.37	0.37
query17	0.94	1.00	0.96
query18	0.21	0.25	0.23
query19	1.92	1.69	1.72
query20	0.01	0.01	0.01
query21	15.45	0.65	0.66
query22	4.24	7.08	2.52
query23	18.27	1.47	1.23
query24	1.94	0.23	0.22
query25	0.14	0.08	0.08
query26	0.26	0.16	0.16
query27	0.09	0.07	0.07
query28	13.31	1.00	0.97
query29	12.61	3.25	3.23
query30	0.25	0.06	0.06
query31	2.84	0.38	0.37
query32	3.28	0.46	0.47
query33	2.83	2.83	2.86
query34	17.19	4.44	4.43
query35	4.53	4.57	4.56
query36	0.64	0.45	0.46
query37	0.18	0.16	0.15
query38	0.16	0.16	0.14
query39	0.05	0.03	0.04
query40	0.17	0.14	0.15
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.68 s
Total hot run time: 30.92 s

Apr 19 '24 13:04 doris-robot

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4cd8438fdb6556c74f6fd8d18fc654d508dbe009 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       14.1 seconds inserted 10000000 Rows, about 709K ops/s

Apr 19 '24 13:04 doris-robot

run buildall

Apr 20 '24 10:04 morningman

TPC-H: Total hot run time: 38393 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d2275d3ce8e9ca44e87ad7a40acb624ffb18373, data reload: false

------ Round 1 ----------------------------------
q1	17601	4299	4237	4237
q2	2002	190	182	182
q3	10443	1187	1228	1187
q4	10186	782	848	782
q5	7502	2701	2619	2619
q6	224	131	134	131
q7	1013	583	583	583
q8	9211	2039	2027	2027
q9	8761	6590	6511	6511
q10	8608	3517	3519	3517
q11	437	242	232	232
q12	466	216	213	213
q13	17795	2927	2971	2927
q14	262	219	225	219
q15	514	475	475	475
q16	501	392	389	389
q17	947	615	659	615
q18	7246	6656	6620	6620
q19	5201	1523	1537	1523
q20	628	310	318	310
q21	3423	2792	2878	2792
q22	356	302	302	302
Total cold run time: 113327 ms
Total hot run time: 38393 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4315	4218	4231	4218
q2	378	268	262	262
q3	2951	2751	2694	2694
q4	1867	1611	1554	1554
q5	5274	5289	5283	5283
q6	204	125	126	125
q7	2262	1856	1877	1856
q8	3214	3335	3367	3335
q9	8589	8528	8547	8528
q10	4053	3856	3939	3856
q11	647	520	500	500
q12	789	614	620	614
q13	17598	3214	3154	3154
q14	330	312	297	297
q15	525	483	485	483
q16	492	459	449	449
q17	1803	1498	1515	1498
q18	8073	7936	7846	7846
q19	1647	1572	1556	1556
q20	2072	1829	1802	1802
q21	5123	5037	4992	4992
q22	553	473	475	473
Total cold run time: 72759 ms
Total hot run time: 55375 ms

Apr 20 '24 10:04 doris-robot

TPC-DS: Total hot run time: 185352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d2275d3ce8e9ca44e87ad7a40acb624ffb18373, data reload: false

query1	894	378	368	368
query2	6163	2695	2397	2397
query3	6660	202	203	202
query4	25241	21428	21223	21223
query5	4148	403	404	403
query6	275	195	171	171
query7	4580	290	286	286
query8	243	200	181	181
query9	8450	2350	2336	2336
query10	418	250	248	248
query11	14926	14114	14151	14114
query12	131	93	81	81
query13	1626	363	358	358
query14	9434	7998	7624	7624
query15	263	187	185	185
query16	8181	262	270	262
query17	1932	569	553	553
query18	2106	277	277	277
query19	332	154	154	154
query20	90	84	88	84
query21	203	131	133	131
query22	5008	4891	4812	4812
query23	33635	33161	33274	33161
query24	7159	3143	3147	3143
query25	616	429	461	429
query26	710	170	172	170
query27	2224	385	381	381
query28	5219	2079	2065	2065
query29	890	659	625	625
query30	313	178	179	178
query31	1026	790	776	776
query32	92	59	56	56
query33	521	259	261	259
query34	883	499	500	499
query35	791	743	705	705
query36	1058	940	926	926
query37	113	74	76	74
query38	3449	3309	3276	3276
query39	1625	1582	1584	1582
query40	176	129	130	129
query41	49	46	44	44
query42	107	108	99	99
query43	602	545	554	545
query44	1097	763	747	747
query45	299	279	278	278
query46	1096	736	764	736
query47	2035	1926	1936	1926
query48	380	307	310	307
query49	851	406	485	406
query50	776	397	398	397
query51	7070	6833	6891	6833
query52	98	87	85	85
query53	343	272	285	272
query54	268	232	236	232
query55	83	71	74	71
query56	242	217	218	217
query57	1205	1100	1123	1100
query58	226	196	204	196
query59	3278	3128	3146	3128
query60	248	224	225	224
query61	87	83	83	83
query62	591	432	439	432
query63	308	275	275	275
query64	4732	4076	3673	3673
query65	3048	3003	3020	3003
query66	743	333	325	325
query67	15767	15159	15095	15095
query68	9053	540	537	537
query69	541	305	310	305
query70	1289	1168	1180	1168
query71	1451	1260	1268	1260
query72	6493	2605	2415	2415
query73	1135	324	320	320
query74	6859	6397	6304	6304
query75	4054	2636	2626	2626
query76	4880	979	1024	979
query77	598	265	298	265
query78	11091	10207	10090	10090
query79	12440	520	502	502
query80	2072	421	439	421
query81	525	239	238	238
query82	783	95	98	95
query83	203	163	163	163
query84	261	82	82	82
query85	1268	261	254	254
query86	406	291	307	291
query87	3501	3259	3237	3237
query88	4983	2320	2319	2319
query89	533	371	369	369
query90	1969	178	182	178
query91	165	95	97	95
query92	57	46	46	46
query93	6997	496	492	492
query94	1203	174	181	174
query95	402	300	305	300
query96	595	262	263	262
query97	3145	2922	2902	2902
query98	236	210	214	210
query99	1280	897	840	840
Total cold run time: 297901 ms
Total hot run time: 185352 ms

Apr 20 '24 10:04 doris-robot

ClickBench: Total hot run time: 29.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d2275d3ce8e9ca44e87ad7a40acb624ffb18373, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.48
query6	1.48	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.56	0.57	0.53
query11	0.16	0.12	0.12
query12	0.15	0.11	0.11
query13	0.60	0.58	0.57
query14	0.75	0.77	0.79
query15	0.83	0.81	0.81
query16	0.36	0.37	0.36
query17	1.03	0.98	1.03
query18	0.20	0.23	0.24
query19	1.80	1.70	1.75
query20	0.01	0.02	0.01
query21	15.39	0.66	0.66
query22	4.59	8.31	1.17
query23	18.36	1.37	1.18
query24	1.51	0.28	0.27
query25	0.15	0.08	0.08
query26	0.28	0.15	0.16
query27	0.08	0.07	0.08
query28	13.39	0.99	0.98
query29	12.58	3.28	3.24
query30	0.28	0.08	0.08
query31	2.83	0.38	0.37
query32	3.27	0.47	0.46
query33	2.78	2.85	2.81
query34	17.31	4.42	4.46
query35	4.44	4.47	4.45
query36	0.65	0.47	0.45
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.19	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 29.56 s

Apr 20 '24 11:04 doris-robot

run buildall

Apr 22 '24 15:04 morningman

run buildall

Apr 23 '24 09:04 morningman

ClickBench: Total hot run time: 31.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12159435565eb90bae593651690720438d320ae5, data reload: false

query1	0.03	0.04	0.03
query2	0.09	0.04	0.04
query3	0.24	0.06	0.06
query4	1.66	0.09	0.08
query5	0.51	0.52	0.50
query6	1.30	0.86	0.82
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.49	0.46	0.44
query10	0.51	0.51	0.51
query11	0.15	0.11	0.12
query12	0.14	0.12	0.11
query13	0.64	0.64	0.64
query14	0.94	0.90	0.98
query15	0.87	0.85	0.85
query16	0.35	0.36	0.37
query17	1.05	1.06	1.05
query18	0.20	0.27	0.21
query19	1.77	1.86	1.80
query20	0.02	0.02	0.01
query21	15.44	0.67	0.66
query22	4.07	7.14	1.97
query23	18.31	1.34	1.30
query24	1.69	0.28	0.30
query25	0.16	0.09	0.10
query26	0.27	0.17	0.18
query27	0.09	0.10	0.09
query28	13.26	1.02	1.01
query29	12.68	3.38	3.37
query30	0.25	0.07	0.07
query31	2.84	0.40	0.40
query32	3.23	0.49	0.49
query33	2.69	2.78	2.78
query34	17.22	4.56	4.61
query35	4.57	4.62	4.57
query36	0.65	0.46	0.47
query37	0.22	0.18	0.19
query38	0.19	0.19	0.19
query39	0.05	0.05	0.05
query40	0.19	0.15	0.14
query41	0.10	0.06	0.07
query42	0.07	0.06	0.06
query43	0.05	0.05	0.05
Total cold run time: 109.33 s
Total hot run time: 31.57 s

Apr 23 '24 09:04 doris-robot

doris doris copied to clipboard

[opt](catalog) support using loading cache for db/table list in external catalog

Proposed changes

Previous Implementation Logic in Doris with External Catalog

Steps

Advantages and Disadvantages

Modifications in this PR

1. User Interface

2. Enabling use_meta_cache

3. Code Implementation

Further comments

doris
doris copied to clipboard

2. Enabling `use_meta_cache`