dotnet
Perf: template generation repetitively writes/reads same target files
Moved from: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1433562/
Steps to reproduce:
- I'm creating simple C# Console app with VS2022 while running ETL collector. It indicates "ConsoleApp1.csproj" file read about 63 times. Of course it could be required thing to do but could be perf improvement opportunity considering IO calls are expensive, please correct me if I'm wrong.
By looking at code stack it looks it has something to do with AggregateTemplateProvider.cs I have ETL traces file, too big too attach here, please let me know if you need one.
Related Discussion:
Vlada Shubina commented May 2 We decided to do a measurement of perf if the processing will be done in memory stream. If it improves perf, then we might switch to in-memory processing if template file is not bigger than X. For bigger files, we still do on-disk processing.
Erick Yondon commented Nov 29, 2021 If I recall correctly, we better write less often to csproj file, because everytime it writes "project system" have to scan for changes and if applicable try to do restore.
David Karlas commented Nov 26, 2021 (edited) Ok, so what is happening is... Our template processor has some cases where it writes stuff but then realizes it needs to reads it back and even deletes some of it, example of this is:
#if hi
Console.WriteLine("Printed if 'hi' defined");
#endif
So in above example... there is whitespace before #if, the way processor works, it keeps writing stuff as it finds it, but once it reaches #if it realizes, oh wait, that whitespace needs to be deleted too... Hence it reads from target stream(output file) back until it reaches NewLine character...
We probably don't want to change all this logic to gain some perf, most likely what we will do instead is do all this processing into MemoryStream and at end dump it into FileStream...
Phil Henning commented Nov 10, 2021 @Vlada Shubina All the code is in the core template engine. This is all likely required, but will let you address since it is in your code.
Jeff Kelly commented Nov 10, 2021 @Phil Henning
Erick Yondon commented Nov 10, 2021 @Joyce Liang
I'm tagging because you're most recent person who contributed this code path(src/env/TemplateProviders/Package/TemplateProviderSupport/AggregateTemplateProvider.cs), please correct me if I'm wrong and forward to correct area.
Related PR (closed): https://github.com/dotnet/templating/pull/4149
