File in GeneratedMSBuildEditorConfig.editorconfig not included

[jjonescz]

For example:

is_global = true
build_property.CsWin32InputMetadataPaths = D:\NuGet\packages\microsoft.windows.sdk.win32metadata\60.0.34-preview\Windows.Win32.winmd|D:\NuGet\packages\microsoft.windows.wdk.win32metadata\0.11.4-experimental\Windows.Wdk.winmd
build_property.CsWin32InputDocPaths = D:\NuGet\packages\microsoft.windows.sdk.win32docs\0.1.42-alpha\apidocs.msgpack
build_property.EnableAotAnalyzer = true
build_property.EnableSingleFileAnalyzer = true
build_property.EnableTrimAnalyzer = true
build_property.IncludeAllContentForSelfExtract = 
build_property.TargetFramework = net8.0
build_property.TargetPlatformMinVersion = 
build_property.UsingMicrosoftNETSdkWeb = 
build_property.ProjectTypeGuids = 
build_property.InvariantGlobalization = 
build_property.PlatformNeutralAssembly = 
build_property.EnforceExtendedAnalyzerRules = 
build_property._SupportedPlatformList = Linux,macOS,Windows
build_property.RootNamespace = Microsoft.Telemetry
build_property.ProjectDir = D:\source\repos\microsoft\SFC.Telemetry.Contract\main\src\Microsoft.Telemetry\
build_property.EnableComHosting = 
build_property.EnableGeneratedComInterfaceComImportInterop = 
build_property.PolySharpUsePublicAccessibilityForGeneratedTypes = 
build_property.PolySharpIncludeRuntimeSupportedAttributes = 
build_property.PolySharpUseInteropServices2NamespaceForUnmanagedCallersOnlyAttribute = true
build_property.PolySharpExcludeGeneratedTypes = 
build_property.PolySharpIncludeGeneratedTypes =       System.Diagnostics.CodeAnalysis.DisallowNullAttribute,      System.Diagnostics.CodeAnalysis.UnscopedRefAttribute,      System.Runtime.CompilerServices.CallerArgumentExpressionAttribute,      System.Runtime.CompilerServices.CompilerFeatureRequiredAttribute,      System.Runtime.CompilerServices.IsExternalInit,      System.Runtime.CompilerServices.RequiredMemberAttribute,      System.Runtime.CompilerServices.SkipLocalsInitAttribute,      System.Runtime.InteropServices.UnmanagedCallersOnlyAttribute,      System.Runtime.Versioning.SupportedOSPlatformAttribute,    
build_property.PolySharpExcludeTypeForwardedToDeclarations = 

The files under build_property.CsWin32InputMetadataPaths are not included in the complog. That causes the CsWin32 source generator to fail. I guess it's not possible to 100% accurately determine which build properties are file paths, but there could be a heuristic I guess (files which exist would be included and their paths translated).

The build_property.ProjectDir seems also like something that should be translated (this seems to be already tracked by https://github.com/jaredpar/complog/issues/281).

[jaredpar]

The files under build_property.CsWin32InputMetadataPaths are not included in the complog.

Are they passed as arguments to the compiler? If not that seems like they're breakiyng the invariants of the compiler by having an untracked input.

[jjonescz]

So they should probably pass the files as AdditionalFiles, right? Should I file an issue on CsWin32?

[jaredpar]

In theory yes. Are these binary files? In the past we've had trouble with how generators / analyzers can distribute binary files and we haven't found a good solution for them. It might be worth revisiting with them.

It's possible we may need to think about a quirk layer. Basically when we know certain components are in play grab extra files, make extra changes. Not sure I want to go down that route or not but it's someting I've been thinking about.

[jjonescz]

Are these binary files? In the past we've had trouble with how generators / analyzers can distribute binary files and we haven't found a good solution for them. It might be worth revisiting with them.

Yes, those are binary files - I see; those would be awkward/impossible to get via AdditionalTextsProvider. Perhaps we just need a new API in Roslyn.

[jaredpar]

Now that we're talking about it some of the problems are getting paged back in. I believe the primary issue is roslyn 54899. Part of the problem is how we represent them in memory. The compiler model means that we have to eagerly do any API before creating a Compilation object. That means for binary files we need to read them into memory. There was some pushback on us doing that because the generator authors said the files may be too big and putting them in memory would be an issue. Don't see any other way though.

[jjonescz]

The compiler model means that we have to eagerly do any API before creating a Compilation object. That means for binary files we need to read them into memory.

Couldn't those binary inputs be represented just by their file path and timestamp; and loaded lazily when needed?

[jaredpar]

The compiler model is generally that a Compilation is 100% self-contained, immutable and has no I/O. It's not 100% strictly that way as there are edges where we do I/O but that is the intent.

It's possible we could make this configurable. For example in the batch compiler there is a hard assumption that the inputs on disk do not change throughout the course of a compilation. Whether we load it eagerly or lazily there is immaterial. For the IDE they generally want eager loading because they want to be able to control when and how I/O happens (ideally async). Possible in those cases we may need to load eagerly. Or possible the source generator paths already have enough async methods that we can still do it as is.

Guessing we could abstract this by saying that there is a way for AdditionalFiles to expose a Stream when we detect it's a binary. Then at the construction of the type we can specify eager / lazy loading. But this would need a bit of design

[jjonescz]

For example in the batch compiler there is a hard assumption that the inputs on disk do not change throughout the course of a compilation.

Right, I imagined a simple solution where we would record the file timestamps/hashes at the beginning (not load them into memory though) and then when we load them lazily we would check the hashes still match and fail if not.

We could also hard-link shadow-copy the file somewhere (like we do for analyzer loading) to make sure it's not tampered with during build.

For the IDE they generally want eager loading because they want to be able to control when and how I/O happens (ideally async).

Interesting, that makes it more complicated, you're right.