Perf: Avoid hidden closures in Scheduler request loop

[ccastanedaucf]

Fixes

Millions of hidden delegate allocations in hot Scheduler loop.

Context

If you check out a memory profile, you may see a generated class in Scheduler as the top allocated type. Here at 315MB, it's taking up a whopping 12% of total allocations:

Supposedly, this is created in ResumeRequiredWork:

Microsoft.Build.BackEnd.Scheduler+<>c__DisplayClass78_0
  Objects : 8258657
  Bytes   : 330346280

►  98.3%  ResumeRequiredWork • 309.74 MB / 309.74 MB • Microsoft.Build.BackEnd.Scheduler.ResumeRequiredWork(List<T>)

But if you look at ResumeRequiredWork or even try to source map over in dottrace, it brings you here:

foreach (SchedulableRequest request in unscheduledRequests)
{
    ResolveRequestFromCacheAndResumeIfPossible(request, responses);
}

Probably some JIT inlining going on, but if you inspect the IL you'll find the real source:

So we have this 48-byte object that's being created millions of times. And if you see where it's used, you'll notice it as the very first line in the function:

Basically, the compiler is trying to do a favor by creating a single type and allocation for all closures that capture the same or similar sets of variables. But in order to do this, it decides to create the allocation at the very start of the function - regardless of whether any of the functions that use the state instance actually need to run.

This means in order to get rid of the allocation, every anonymous and local function here needs to be rewritten to avoid capturing any local or instance state.  

Changes Made

• Modify all local functions in CheckIfCacheMissOnReferencedProjectIsAllowedAndErrorIfNot to avoid capturing local state
• Add defensive static keyword to local functions (really just syntactic sugar since it won't allocate either way, but hopefully prevents an accidental refactor regressing)