Proposal: Avoid relying on LINQ in product code for improved performance

[kasperk81]

There are approximately 200 occurrences of LINQ usage in MSBuild's non-test and non-sample source code. While LINQ provides clean and expressive syntax, it is not considered high-performance, as noted in discussions such as https://github.com/dotnet/runtime/issues/19382#issuecomment-734524103 and https://github.com/dotnet/runtime/issues/76205. LINQ introduces overhead in certain performance-sensitive scenarios, which can be suboptimal for a product like MSBuild.

The Challenge

Replacing LINQ across MSBuild's codebase is not a trivial task. From my experience, it took five pull requests to remove LINQ from the microsoft/kiota's dependency repositories. A direct replacement would involve significant effort, especially when considering that LINQ expressions are often tightly integrated into the logic.

Proposed Solution: LinqGen

A more practical and less invasive solution is to replace LINQ with LinqGen, an MIT-licensed, source-generated variant of LINQ. LinqGen provides the same expressive syntax as LINQ but eliminates the performance penalties and code bloat associated with runtime LINQ by generating optimized code at compile time using Roslyn source generators.

This approach offers several advantages:

• Minimal Code Changes: LinqGen can often serve as a direct replacement for LINQ, reducing the need for large-scale refactoring.
• Improved Performance: LinqGen addresses the performance issues tied to LINQ's runtime overhead, making it more suitable for a high-performance application like MSBuild.
• Mitigating Code Bloat: Source generation with LinqGen ensures that only the necessary code is generated, avoiding the code bloat commonly associated with LINQ in performance-critical paths.
• NativeAOT Compatibility: Though not a primary goal for MSBuild, it's worth noting that LinqGen offers better compatibility with NativeAOT. This was one of the key reasons I chose this approach for microsoft/kiota.

Conclusion

By switching LINQ usage in MSBuild's product code to LinqGen, we can achieve better performance without invasive changes to the codebase. This change will help optimize MSBuild's efficiency while maintaining readable and expressive code. Additionally, it provides the side benefit of improved NativeAOT compatibility, should that ever become a consideration in the future.