Why the very first design time build after switching from console to VS IDE changes the *.csproj.AssemblyReference.cache files?

[MarkKharitonov]

Visual Studio Version

17.1.3

Summary

I check the *.csproj.AssemblyReference.cache files after console msbuild and after the first design time build in VS IDE. They are different even though nothing has changed. Note that consecutive msbuild runs or regular builds in VS IDE do not change them. Only after switching from the console build to a new VS IDE instance that runs the first design time build do we see the issue.

Steps to Reproduce

1. git clean -qdfx
2. build with msbuild
3. Save the *.csproj.AssemblyReference.cache files
4. build with msbuild
5. Save the *.csproj.AssemblyReference.cache files
6. open devenv
7. wait for the design time build to finish
8. Check the *.csproj.AssemblyReference.cache files

I have the binary logs and the selected files saved aside for each build including the design time build, but I do not want to upload them to a public location. Is there a private upload link?

Expected Behavior

All the files are identical.

Actual Behavior

The design time build produces a different file

User Impact

I do not know. It may be a symptom of a bigger problem.

[drewnoakes]

Could it be that different versions of MSBuild are involved in each of your scenarios? Does it reproduce if you use the VS developer command prompt?

[MarkKharitonov]

We use exactly the same version of msbuild. The $PROFILE script loads the dev tools. Here:

C:\> msbuild -version
Microsoft (R) Build Engine version 17.1.0+ae57d105c for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

17.1.0.7609
C:\> (get-command msbuild).path
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\\MSBuild\Current\Bin\amd64\MSBuild.exe
C:\>

I can upload the binary logs, along with some files I captured after each step. But I prefer a private upload link if possible.

[MarkKharitonov]

https://github.com/dotnet/msbuild/issues/7752

[MarkKharitonov]

Just checked with 17.2.5 - same behavior. The *.csproj.AssemblyReference.cache files are changed by the design time build.

[MarkKharitonov]

I have additional information. I have tested on a big solution (200+ projects). This time some (not all) *.csproj.AssemblyReference.cache files are rewritten after the second msbuild run (when nothing has changed). So, no recompilation, but without knowing the msbuild internals I fear these changes may affect other important (and expensive) targets like ResolveAssemblyReference.

I feel this should be moved to the msbuild github issues.

I have the binary logs. I know having a real repro code is better, but that would have take time I do not have at the moment.

[drewnoakes]

Moving to dotnet/msbuild as I don't see how the project system is involved here.

[danmoseley]

Are the cache files byte for byte identical with the ones they over wrote?

[MarkKharitonov]

No, they are different. The code does not recompile, so it does not affect the Csc task, but I am afraid it affects the ResolveAssemblyReferences task, which is not less expensive. I have captured the timestamps before and after plus the binary logs themselves. I can rerun it and capture some cache files before and after. If there is a private upload link, I can upload all of it there.

[danmoseley]

It might be interesting to binary diff them to get an idea what changed, e.g. something like a version number. (I'm not on the team and no doubt the code is totally changed from when I was)

[rainersigwald]

@MarkKharitonov for a private-to-Microsoft upload channel you can open a feedback ticket. After it's created, that will open an internal bug. If you post the link here we can bypass the usual routing.

This doesn't seem like it should be the case, but I am skeptical that just changing the file is causing any perf degradation. RAR is not completely bypassed when the cache is up to date, so touching it is likely irrelevant.

Theory: the change in the cache may be a result of running on a different node that has built different projects and thus has different things in its in-memory cache.

[MarkKharitonov]

Here you go - https://developercommunity.visualstudio.com/t/Why-msbuild-overwrites-some-csprojAss/10090995?space=61&entry=problem

How do I get an upload link?

[rainersigwald]

@MarkKharitonov You should be able to add a comment, mark it as private to Microsoft, and use the paper-clip icon to add arbitrary files.

[MarkKharitonov]

I was able to upload all the logs to the aforementioned feedback issue. Thank you.

[Forgind]

Theory: the change in the cache may be a result of running on a different node that has built different projects and thus has different things in its in-memory cache.

If the in-memory cache has anything the file state doesn't, it should write it into the file state, so the file state should be complete either way. Guess pre-looking at it: danmoseley's version change sounds plausible, or it could just be that it found the same version at a different path.

[Forgind]

Small update: I arbitrarily looked at the second assembly that was "different." Apparently the version was the same, but the lastModified timestamp changed—from 3/26/2018 to 3/27/2018. This was Microsoft.CSharp.dll. The paths were the same, as was everything else about them. My current hypothesis is that there's something different about how timestamps are calculated between when we calculate them from a command line build and when VS calculates them. I can probably find how we calculate them, but I will need to solicit help to figure out how VS calculates them.

[Forgind]

The two timestamps are four hours apart. @MarkKharitonov, are you four hours away from UTC, by chance? (In either direction)

[Forgind]

Ok, so we seem to use UTC. I would argue that's better behavior, so if I can find code suggesting VS uses local time for timestamps, I'd suggest we recommend they change that. (I'm not sure if they'll be able to; it might be a breaking change.)

[Forgind]

The other interesting part of this is that it only seemed to affect some packages. If I'm right, that suggests that some packages' timestamps were recorded in (or converted to) local time, whereas others were UTC.

[MarkKharitonov]

@Forgind - I am in EST, which is indeed 4 hours off the UTC.

[danmoseley]

I see no reason for time stamps to be stored in local time. When I travel, it shouldn't trigger a clean build.

One scenario VS might care about though (assuming it is a VS issue) is operating on the same build tree with both an older and newer VS. Presumably switching to the other one should not trigger a rebuild either.