doris icon indicating copy to clipboard operation
doris copied to clipboard

[Bug](profile) fix the BytesScannedFromCache profile always is 0

Open zhangstar333 opened this issue 2 weeks ago • 5 comments

What problem does this PR solve?

Problem Summary: the cache_profile will call update in _collect_profile_before_close() function, so couldn't reset about bytes_read_from_local/remote fields.

void FileScanner::_collect_profile_before_close() {
    Scanner::_collect_profile_before_close();
    if (config::enable_file_cache && _state->query_options().enable_file_cache &&
        _profile != nullptr) {
        io::FileCacheProfileReporter cache_profile(_profile);
        cache_profile.update(_file_cache_statistics.get());
        _state->get_query_ctx()->resource_ctx()->io_context()->update_bytes_write_into_cache(
                _file_cache_statistics->bytes_write_into_cache);
    }

Release note

None

Check List (For Author)

  • Test

    • [ ] Regression test
    • [ ] Unit Test
    • [ ] Manual test (add detailed scripts or steps below)
    • [ ] No need to test or manual test. Explain why:
      • [ ] This is a refactor/code format and no logic has been changed.
      • [ ] Previous test can cover this change.
      • [ ] No code files have been changed.
      • [ ] Other reason
  • Behavior changed:

    • [ ] No.
    • [ ] Yes.
  • Does this need documentation?

    • [ ] No.
    • [ ] Yes.

Check List (For Reviewer who merge this PR)

  • [ ] Confirm the release note
  • [ ] Confirm test cases
  • [ ] Confirm document
  • [ ] Add branch pick label

zhangstar333 avatar Dec 11 '25 09:12 zhangstar333

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Thearas avatar Dec 11 '25 09:12 Thearas

run buildall

zhangstar333 avatar Dec 11 '25 12:12 zhangstar333

TPC-H: Total hot run time: 36607 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59bbc637e365e843fd32f24f872a3a44714ace80, data reload: false

------ Round 1 ----------------------------------
q1	17605	4219	4077	4077
q2	2009	348	226	226
q3	10186	1366	733	733
q4	10232	897	314	314
q5	7528	2122	1931	1931
q6	184	165	136	136
q7	1020	854	717	717
q8	9351	1402	1197	1197
q9	7028	5283	5374	5283
q10	6846	2380	1953	1953
q11	504	317	293	293
q12	665	719	588	588
q13	18220	3655	3103	3103
q14	303	325	283	283
q15	596	513	515	513
q16	940	942	888	888
q17	716	855	511	511
q18	7553	7977	7742	7742
q19	1878	1058	638	638
q20	407	394	259	259
q21	4555	4213	4212	4212
q22	1141	1083	1010	1010
Total cold run time: 109467 ms
Total hot run time: 36607 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4293	4241	4241
q2	336	499	345	345
q3	2593	2777	2405	2405
q4	1410	1846	1401	1401
q5	4631	4466	4462	4462
q6	216	164	123	123
q7	2169	1897	1766	1766
q8	2623	2495	2640	2495
q9	7569	7418	7446	7418
q10	2879	3099	2645	2645
q11	564	497	483	483
q12	645	671	537	537
q13	3220	3655	2989	2989
q14	257	290	256	256
q15	531	494	496	494
q16	850	885	834	834
q17	1115	1348	1368	1348
q18	7281	7009	6962	6962
q19	825	811	812	811
q20	1897	1975	1819	1819
q21	4656	4268	4172	4172
q22	1121	999	985	985
Total cold run time: 51750 ms
Total hot run time: 48991 ms

doris-robot avatar Dec 11 '25 12:12 doris-robot

TPC-DS: Total hot run time: 181008 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59bbc637e365e843fd32f24f872a3a44714ace80, data reload: false

query5	5062	628	462	462
query6	326	240	209	209
query7	4222	467	277	277
query8	322	250	230	230
query9	8775	2562	2565	2562
query10	541	382	318	318
query11	15226	14772	14882	14772
query12	181	114	111	111
query13	1250	508	401	401
query14	6276	3266	3009	3009
query14_1	2936	2899	2878	2878
query15	208	202	186	186
query16	973	484	449	449
query17	1129	723	603	603
query18	2548	449	357	357
query19	235	263	209	209
query20	122	117	111	111
query21	219	140	121	121
query22	3899	3969	4015	3969
query23	16491	16162	15986	15986
query23_1	15982	15962	16059	15962
query24	7395	1688	1276	1276
query24_1	1248	1220	1249	1220
query25	592	500	450	450
query26	1261	276	166	166
query27	2750	471	323	323
query28	4469	2161	2144	2144
query29	814	582	466	466
query30	315	251	212	212
query31	854	685	631	631
query32	90	69	68	68
query33	557	344	302	302
query34	904	918	549	549
query35	809	804	730	730
query36	890	956	814	814
query37	128	96	135	96
query38	3848	3852	3764	3764
query39	757	730	719	719
query39_1	698	707	698	698
query40	223	140	120	120
query41	66	67	62	62
query42	107	103	109	103
query43	434	430	410	410
query44	1371	760	759	759
query45	195	189	185	185
query46	891	981	630	630
query47	1657	1687	1626	1626
query48	319	330	244	244
query49	667	425	354	354
query50	670	296	220	220
query51	3840	3807	3797	3797
query52	113	119	98	98
query53	328	371	298	298
query54	295	261	255	255
query55	80	78	72	72
query56	306	288	299	288
query57	1150	1142	1096	1096
query58	263	251	250	250
query59	2465	2443	2325	2325
query60	308	315	293	293
query61	201	153	157	153
query62	721	702	625	625
query63	324	298	299	298
query64	4930	1299	988	988
query65	4023	3971	3912	3912
query66	1419	444	322	322
query67	15087	14912	14695	14695
query68	8334	1042	738	738
query69	494	346	308	308
query70	1084	977	980	977
query71	366	315	280	280
query72	6038	4950	5051	4950
query73	660	568	316	316
query74	8833	8933	8588	8588
query75	3606	3538	3152	3152
query76	3944	1136	757	757
query77	528	394	297	297
query78	9406	9568	8847	8847
query79	1443	888	601	601
query80	733	655	554	554
query81	512	268	229	229
query82	203	131	111	111
query83	269	256	241	241
query84	265	116	95	95
query85	913	497	449	449
query86	386	288	293	288
query87	4020	4091	3982	3982
query88	3178	2277	2278	2277
query89	456	429	396	396
query90	2164	165	159	159
query91	169	168	140	140
query92	86	68	63	63
query93	2036	917	571	571
query94	475	313	285	285
query95	578	373	299	299
query96	594	464	211	211
query97	2568	2648	2542	2542
query98	213	197	194	194
query99	1305	1269	1216	1216
Total cold run time: 264073 ms
Total hot run time: 181008 ms

doris-robot avatar Dec 11 '25 13:12 doris-robot

ClickBench: Total hot run time: 27.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59bbc637e365e843fd32f24f872a3a44714ace80, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.12	0.10
query5	0.28	0.27	0.28
query6	1.17	0.64	0.62
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.51	0.50
query10	0.57	0.56	0.55
query11	0.16	0.11	0.12
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.98	0.98	0.99
query15	0.82	0.82	0.79
query16	0.42	0.41	0.39
query17	1.10	1.09	1.04
query18	0.24	0.21	0.21
query19	1.97	1.86	1.83
query20	0.02	0.02	0.01
query21	15.42	0.30	0.14
query22	4.83	0.06	0.04
query23	16.08	0.28	0.10
query24	1.60	1.26	0.29
query25	0.10	0.08	0.05
query26	0.15	0.14	0.14
query27	0.07	0.06	0.05
query28	4.26	1.23	1.02
query29	12.59	3.97	3.22
query30	0.27	0.13	0.12
query31	2.83	0.63	0.40
query32	3.23	0.56	0.45
query33	3.05	2.99	3.02
query34	16.90	5.17	4.53
query35	4.58	4.63	4.54
query36	0.68	0.50	0.49
query37	0.10	0.06	0.06
query38	0.07	0.04	0.03
query39	0.05	0.03	0.03
query40	0.17	0.15	0.13
query41	0.09	0.03	0.04
query42	0.04	0.02	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.4 s
Total hot run time: 27.27 s

doris-robot avatar Dec 11 '25 13:12 doris-robot

BE UT Coverage Report

Increment line coverage :tada:

Increment coverage report Complete coverage report

Category Coverage
Function Coverage 53.42% (18822/35237)
Line Coverage 39.16% (174036/444401)
Region Coverage 33.83% (134855/398641)
Branch Coverage 34.75% (58008/166910)

doris-robot avatar Dec 12 '25 03:12 doris-robot