parquet-dotnet icon indicating copy to clipboard operation
parquet-dotnet copied to clipboard

Performance slowdown 20x between 3.2.0 and master

Open ay89607 opened this issue 3 years ago • 0 comments

Hi! I'm trying to use the library for our project and found that master branch performance is much slower vs. 3.2.0 (just took older version randomly), especially for GZip compression.

I ran both tests on the same machine, master on VS 2019 (Release/Any CPU), 3.2.0 on VS2017 (Release/Any CPU), Ctrl-F5 - just 2 different repos, no code changes.

Here is results from Parquet.Runner for 3.2.0:

0:40:55,972|INF|Parquet.Runner.Program|iteration #0: 00:00:00.2031456, uwp: 00:00:00.0994359, gwt: 00:00:00.1457194 0:40:56,345|INF|Parquet.Runner.Program|iteration #1: 00:00:00.1771764, uwp: 00:00:00.0418373, gwt: 00:00:00.1331094 0:40:56,679|INF|Parquet.Runner.Program|iteration #2: 00:00:00.1514195, uwp: 00:00:00.0423653, gwt: 00:00:00.1344789 0:40:57,012|INF|Parquet.Runner.Program|iteration #3: 00:00:00.1546678, uwp: 00:00:00.0447164, gwt: 00:00:00.1292321 0:40:57,355|INF|Parquet.Runner.Program|iteration #4: 00:00:00.1451514, uwp: 00:00:00.0400017, gwt: 00:00:00.1530406 0:40:57,680|INF|Parquet.Runner.Program|iteration #5: 00:00:00.1384282, uwp: 00:00:00.0567719, gwt: 00:00:00.1253477 0:40:57,980|INF|Parquet.Runner.Program|iteration #6: 00:00:00.1262184, uwp: 00:00:00.0387629, gwt: 00:00:00.1299687 0:40:58,312|INF|Parquet.Runner.Program|iteration #7: 00:00:00.1518486, uwp: 00:00:00.0446676, gwt: 00:00:00.1302474 0:40:58,645|INF|Parquet.Runner.Program|iteration #8: 00:00:00.1448375, uwp: 00:00:00.0510432, gwt: 00:00:00.1321102 0:40:59,002|INF|Parquet.Runner.Program|iteration #9: 00:00:00.1638344, uwp: 00:00:00.0560278, gwt: 00:00:00.1316949 0:40:59,008|INF|Parquet.Runner.Program|mean(read): 00:00:00.1556727, mean(uw): 00:00:00.0515630, mean(gw): 00:00:00.1344949

Here is results from Parquet.Runner for master:

iteration #0: 00:00:00.3910547, uwp: 00:00:00.3726689, gwt: 00:00:02.4269554 iteration #1: 00:00:00.2854977, uwp: 00:00:00.2408480, gwt: 00:00:02.1413147 iteration #2: 00:00:00.2963417, uwp: 00:00:00.2690069, gwt: 00:00:02.3192123 iteration #3: 00:00:00.2569918, uwp: 00:00:00.2376890, gwt: 00:00:02.0613361 iteration #4: 00:00:00.3288451, uwp: 00:00:00.2519022, gwt: 00:00:02.2089186 iteration #5: 00:00:00.2533449, uwp: 00:00:00.2269109, gwt: 00:00:02.0503941 iteration #6: 00:00:00.2645899, uwp: 00:00:00.2278158, gwt: 00:00:02.0869678 iteration #7: 00:00:00.2557560, uwp: 00:00:00.2440898, gwt: 00:00:02.0265803 iteration #8: 00:00:00.2819648, uwp: 00:00:00.2333260, gwt: 00:00:02.1166082 iteration #9: 00:00:00.2821608, uwp: 00:00:00.2289063, gwt: 00:00:02.0486098 mean(read): 00:00:00.2896547, mean(uw): 00:00:00.2533163, mean(gw): 00:00:02.1486897

ay89607 avatar Mar 04 '21 00:03 ay89607