Help with RazorExtension.dll failing SymbolCheck

[davidwengier]

Build

Razor insertions are failing SymbolCheck for one specific DLL, and I've tried to dig in to find out why, but I have reached my limit, and would appreciate some guidance or assistance.

eg, on our most recent insertion: https://devdiv.visualstudio.com/DevDiv/_git/VS/pullrequest/420798

SymbolCheck failed for microsoft.visualstudio.razorextension.dll https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=6631089&view=logs&j=fd490c07-0b22-5182-fac9-6d67fe1e939b&t=da4239c9-cabf-50a7-e7ce-52be5c6d9883&s=96ac2280-8cb4-5df5-99de-dd2da759617d (Make sure you look at Attempt 1 in the build log. I tried to re-run the build out of curiosity, but it failed because the PR had already been merged and branch gone)

SymbolCheck issue: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1601388/

Build leg reported

SymbolCheck

Pull Request

https://devdiv.visualstudio.com/DevDiv/_git/VS/pullrequest/420798

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• [ ] Open the failing build above and investigate
• [ ] Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "There are 1 unarchived files",
   "BuildRetry": false
}

Additional information about the issue reported

I'm not even sure where to look for logs/artifacts for the symbol publishing part of our CI build so even that information could be useful, but if you want to go all the way and open a PR, I'll accept that too :D

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[andriipatsula]

Hello @davidwengier , looking into this.

[Chrisboh]

@andriipatsula what did you find out here / what actions should be taken?

[Chrisboh]

In looking at this more we don't own symbol check. @mikem8361 do you know how David could best investigate this?

[mikem8361]

@hoyosjs

[Chrisboh]

@hoyosjs do you have a few minutes to help out?

[hoyosjs]

I looked at this a little bit. It looks like there's no package that contains Microsoft.VisualStudio.RazorExtension.pdb. Arcade ends up publishing it from the artifacts in the pipeline that you can see at PDBArtifacts\Microsoft.VisualStudio.RazorExtension\net472\Microsoft.VisualStudio.RazorExtension.pdb. However, that PDB doesn't match the Microsoft.VisualStudio.RazorExtension.pdb checksum that I see in the debug directory section of the DLL that's in the VSIX, so when VS symbol check passed over that dll it fails. I see there's one in FinalOutputPath = D:\a\_work\1\s\artifacts\bin\Microsoft.VisualStudio.RazorExtension\Release\net472\Microsoft.VisualStudio.RazorExtension.pdb and one in FinalOutputPath = D:\a\_work\1\s\src\Razor\src\Microsoft.VisualStudio.RazorExtension\Microsoft.VisualStudio.RazorExtension.pdb , the later one coming from a generate project in GenerateTemporaryTargetAssembly which comes from the windows desktop SDK. It uploading one that doesn't match is a possibility. A double write + non-determinism is another one. Essentially, what needs to be investigated is where does the PDB for the DLL that ends up in the VSIX is and why it doesn't match the one in PDBArtifacts.

[ulisesh]

@hoyosjs, thanks for taking a looking

Closing as this isn't a problem with our infrastructure given that there is a mismatch between the dll and its pdb

[davidwengier]

Closing as this isn't a problem with our infrastructure given that there is a mismatch between the dll and its pdb

@ulisesh does arcade not control the build scripts that would produce these pdbs? Are there any pointers for how I might investigate "where does the PDB for the DLL that ends up in the VSIX is and why it doesn't match the one in PDBArtifacts."? I don't think our csproj files contain anything that would affect that, so I assume if I go digging through binlogs I'm just going to run into arcade concepts straight away. If there is no better option than that, though, let me know.

[ulisesh]

@hoyosjs from what I understood from your investigation the symbol mismatch isn't caused by a problem with arcade, did I miss something?

[hoyosjs]

I looked a bit at this in the razor-tooling build a bit more. There's a target - DeployToSymStore - in arcade that does some portable PDB to full PDB translation per DevDiv policy. The output of this is the one that ends up in the PDBsToPublish group:

Copying file from "D:\a_work\1\s\artifacts\SymStore\Release\Microsoft.VisualStudio.RazorExtension\net472\Microsoft.VisualStudio.RazorExtension.pdb" to "D:\a_work\1\s\artifacts\tmp\Release\PDBsToPublish\Microsoft.VisualStudio.RazorExtension\net472\Microsoft.VisualStudio.RazorExtension.pdb".

However the one in the VSIX doesn't match that PDB and hasn't gone through the PDB conversion process (It'd have an extra entry in the debug directory).

  Debug Directories

        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    97B5277D cv            88 0007B134    79334    Format: RSDS, {CA50C668-4D8E-4BA6-9E38-66DB25DFA1AE}, 1, /_/artifacts/obj/Microsoft.VisualStudio.RazorExtension/Release/net472/Microsoft.VisualStudio.RazorExtension.pdb
    00000000 pdbhash       27 0007B1BC    793BC    SHA256: 68 C6 50 CA 8E 4D A6 6B 5E 38 66 DB 25 DF A1 AE 7D 27 B5 17 43 A6 AE 05 5C 02 35 FF 9E D9 8B E3
    00000000 repro          0 00000000        0

The main problem is you call build twice. Once in the regular build pass, where it emits the PDB and does the conversion. Then a second pass to build the VSIX which retriggers the Build target on the project and reruns CoreCompile since CleanupTemporaryTargetAssembly deletes "D:\a\_work\1\s\artifacts\obj\Microsoft.VisualStudio.RazorExtension\Release\net472\Microsoft.VisualStudio.RazorExtension.dll" does not exist. which is part of the outputs. You end up with 2 different DLLs. One with a DLL + PDB for nupkgs, and one associated with the VSIX DLL. The first pass runs PublishToAzureDevOpsArtifacts (consequence of the -publish flag to build) which publishes the PDB for indexing. The second pass doesn't do this so that PDB ends up not matching. Even if it tried to publish (which has its own problem since you are doing double writes) AzDO will fail to publish an asset with the same name. There are a few mitigations, but in general these double writes seem to be causing more issues than solve problems it looks like.

[davidwengier]

Thanks for the detailed info @hoyosjs, in particular "you call build twice" helped me work out where to look :)