MSBuildStructuredLog
MSBuildStructuredLog copied to clipboard
KirillOsenkov
Is it possible to generate a "truncated log" optimized for finding incremential build failures?
We are dealing with very large solution files (~3100 projects) the log that is generated in these scenarios is quite large (30GB+) opening them up is difficult in MSBuildStructuredLog, as is attempting to filter.
We are attempting to understand why we lose incremental builds; would it be possible to get a truncated log within MSBuildStructuredLog viewer to at least point us towards the project that is triggering the rebuild? From there we can work on narrowing down the root cause.
We need something akin to Visual Studio's "Project XXXX was out of date due to YYYY Rebuilding". I think I have had discussions with you as to why this is not possible in MSBuild (its that Visual Studio uses a slightly different up-to-date check which can give you that information). We have found your posts on enabling U2DCheckVerbosity, which is the path we're going down now, but wanted to throw this out there.
Wow, the binlog is 30GB or is the equivalent text log 30 GB? Max I've seen is 1-2 GB binlogs and they expand to 30-50 GB of text.
We do have a lot of opportunity to significantly reduce binlog sizes: https://github.com/KirillOsenkov/MSBuildStructuredLog/wiki/BinLog-Perf
You can search for strings that contain one of:
Building target completelyis newer than outputout-of-date
You can probably write a simple tool that calls the raw ReadRecords() API, reads all the strings and prints out the strings that contain these substrings. It might give you some hints. But it's all a heuristic. I'm imagining there might be an algorithm that detects the "primary origin" of the rebuild, to filter out cascading subsequent rebuilds. But it'll need a lot of thinking and a lot o heuristics. Still something to think about.
@KirillOsenkov Well I might have exaggerated a bit; it looks like it is 21.5GB, still bad times bro:
Wow this is nuts! I assume there's no way you could share that binlog privately with me? Would be very useful to dissect it and see where most of the space goes. Although I'll probably have to allocate an Azure VM with 256 GB RAM to even open it. How does it open at all? My first name dot last name at my employer is the email.
Although I'll probably have to allocate an Azure VM with 256 GB RAM to even open it.
I've got an F64s_v2 instance and it struggles (64 vcpus, 128 GiB memory) that being said I did cannibalize 64gb of that RAM to get a RAM Drive to get any type of perf out of this when building (takes about 10 minutes to build in this configuration, I feel bad for anyone else on this shared node because there is no rest for those cores).
Keep a look out in that inbox.
If you happen to have the monster open in the binlog viewer, what's the memory consumption of StructuredLogViewer.exe? I need Virtual Memory, Commit and Private Bytes.
@KirillOsenkov about 30 minutes:
The bin logger totally killed my build times too, that should have taken less than 20 minutes on that hardware, the overhead was massive, but at this time I am not interested in chasing that down, the incremental build is the focus.
Yes, the overhead is unfortunately in line with what we're seeing on other large solutions. We know exactly what to fix and what to optimize, it just needs time. Oh how I wish I had more time to work on fixing this.
Oh how I wish I had more time to work on fixing this
You and me both buddy, its amazing I have gotten it this far.