Perfcollect, dotnet 6 & crossgen2

[kkostrzewa]

I'm trying to run perfcollect on linux (WSL Ubuntu 20.4) to collect native traces and I'm getting the warning Crossgen not found. Framework symbols will be unavailable.

I see perfcollect is dependent on crossgen, which doesn't appear to exist on dotnet 6. It looks like that's been replaced with crossgen2

Is there an updated version of perfcollect that uses crossgen2 instead?

[brianrob]

@kkostrzewa, I just installed WSL so that I could test this, but I can't seem to get perf to run, because there isn't a version of perf that matches the kernel version. Did you build perf from source, or did you otherwise workaround this?

[kkostrzewa]

@brianrob yes, I built perf from source following the steps in https://stackoverflow.com/a/60276918/3794464

[brianrob]

@kkostrzewa the design is that for .NET 5+ you don't need crossgen to generate framework symbols because we use the perf JIT dump feature instead. It looks as though the warning logic isn't working properly because it's not finding the jit dump file that is being created. I'm also having trouble running perfcollect as is on my setup, and so I can't tell if the symbols are showing up properly. Are you by chance able to share a sample trace file? I'm interested to see how far you can get, and what your trace file looks like.

[kkostrzewa]

@brianrob I haven't forgotten about this - work has me blocked on getting you the data that you need. Thanks for your time.

[brianrob]

@kkostrzewa, no worries. Thanks for the heads-up.

[AloisKraus]

I have the same issue that running a simple Console app on Ubuntu does not lead to stack traces for JIT compiled code. I have also tried ReadyToRun images but that did also not lead to stacks for my code.

Console.WriteLine("Hello, World!");
while(true)
{
    
}

dotnet publish -c Release -r linux-x64 -p:PublishReadytoRun=true

lois@aloisvm:~/bin$ sudo ./perfcollect.sh  collect start -threadtime 
Collection started. Press CTRL+C to stop.
^C
...STOPPED.

Starting post-processing. This may take some time.

Generating native image symbol files
...SKIPPED
Crossgen not found. Framework symbols will be unavailable.
See https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md#resolving-framework-symbols for details.
...FINISHED
Saving native symbols
...FINISHED
Resolving JIT and R2R symbols
...FINISHED
Exporting perf.data file
...FINISHED
Compressing trace files
...FINISHED
Cleaning up artifacts
...FINISHED

Trace saved to start.trace.zip

After tracing has started I run ./MyApp which goes into an endless loop and then terminate it via Ctrl-C.

The resulting trace is

https://onedrive.live.com/?authkey=%21AJIusb6cHxwUCYk&id=89CCF7CEB5AB0517%2143684&cid=89CCF7CEB5AB0517

[brianrob]

@AloisKraus, did you set COMPlus_PerfMapEnabled=true for the process being profiled? If not, this will turn on JIT and R2R symbol generation.

[xiaofeng-zheng]

        # Get the list of loaded images and use the path to libcoreclr.so to find crossgen.
        # crossgen is expected to sit next to libcoreclr.so.
        local buildidList=`$perfcmd buildid-list | grep libcoreclr.so | cut -d ' ' -f 2`
        local crossgenCmd=''
        local crossgenDir=''
        for file in $buildidList
        do
            crossgenDir=`dirname "${file}"`
            if [ -f ${crossgenDir}/crossgen ]
            then
                crossgenCmd=${crossgenDir}/crossgen
                LogAppend "Found crossgen at ${crossgenCmd}"
                break
            fi
        done

crossgen is hard coded in perfcollect.

[brianrob]

crossgen is hardcoded because we should not need to use crossgen or crossgen2 starting with .NET 5. Instead, we use a different mechanism to generate these symbols which is controlled via COMPlus_PerfMapEnabled.

[xiaofeng-zheng]

Actually I tried this. It seemed not to work. But I'll do it again to double-check.

[brianrob]

Checking in to see if you folks are still running into issues with this. Otherwise, I'll plan to close this. Thanks.

[pcwiese]

I haven't been able to successfully use perfcollect at all since moving to net6. As a sanity check, I created a boilerplate net6 console app using

dotnet new console -n PerfcollectTest-f net6.0

add a Console.ReadLine() to Program.cs add a Dockerfile where I just copy in the publish dir output

FROM mcr.microsoft.com/dotnet/aspnet:6.0

WORKDIR /app
COPY ./ ./

ENTRYPOINT ["dotnet", "PerfcollectTest.dll"]

build and pack it

dotnet publish;docker build -f .\Dockerfile Z:\src\pwiese\private\PerfcollectTest\bin\Debug\net6.0\publish\ -t perfcollecttest

Run thing in docker desktop set to linux containers

docker run -it --name perfcollecttest -e DOTNET_PerfMapEnabled=1 -e DOTNET_EnableEventLog=1 perfcollecttest

From another window, exec into its bash shell, install perfcollect, and try a simple capture

PS C:\Users\pwiese> docker exec -it perfcollecttest bash
root@63a2785b2f64:/app# apt update;apt install -y curl;curl -OL https://aka.ms/perfcollect;chmod +x perfcollect;./perfcollect install

root@63a2785b2f64:/app# ./perfcollect collect sampleTrace
Collection started. Press CTRL+C to stop.
^C./perfcollect: line 702: kill: (1997) - No such process

...STOPPED.

Starting post-processing. This may take some time.

/usr/bin/perf: line 13: exec: perf_5.10.102: not found
E: linux-perf-5.10.102 is not installed.
Generating native image symbol files
/usr/bin/perf: line 13: exec: perf_5.10.102: not found
E: linux-perf-5.10.102 is not installed.
...FINISHED
Saving native symbols
/usr/bin/perf: line 13: exec: perf_5.10.102: not found
E: linux-perf-5.10.102 is not installed.
...FINISHED
Resolving JIT and R2R symbols
...FINISHED
Exporting perf.data file
...FINISHED
Compressing trace files
...FINISHED
Cleaning up artifacts
...FINISHED

Trace saved to sampleTrace.trace.zip

It doesn't properly find or install perf and the capture is empty.

[brianrob]

@pcwiese, what you're hitting here is a problem that is specific to containers - specifically where the container image doesn't match the host distro. Perf_events is built to be kernel version specific, because it is part kernel-space, part user-space. If you run perf, it's actually a shell script that looks for the binary that matches the currently running kernel, and will fail in this way if it can't find it. Because different distros rev their kernel versions at different times and to different versions, this sort of thing can happen. Perfcollect does its best to work around this and find a close version to run, but the support for this is currently specific to Ubuntu. I am looking to add support for Debian, which will address this.

[AloisKraus]

@brianrob: What is the solution to this if this happens? Download perf by yourself and rename it to the version perfcollect is expecting, or will this not work because of differing kernel versions? Is it possible that you cannot profile at all even if you copy all the tools manually?

[brianrob]

In this case, the best option is to try and use a different version of the perf tool. The interface between the kernel and user portions of perf is fairly stable, and the intention as stated by the maintainers is to make it possible to run mismatched versions.

In perfcollect, we try to select any version that is available and use that. I'm part way through the implementation for Debian. Otherwise, yes, if you wanted to match exactly, you might have to build it yourself and edit perfcollect to use your private copy. I have done this in the past to test bug fixes to perf. It's possible, though definitely not something I'd like to ask most people to do.

[AloisKraus]

@brianrob: Thanks! So there are options. This creates impact since we want to profile but if I need to ensure that the images are compatible with perfcollect things become more complex because it limits which images we can use.

[brianrob]

Yes, there are options. We do our best to handle these cases in perfcollect, but as you can see it's definitely not perfect. I've just posted #1737, which will fix this for Debian, and should unblock you.

[brianrob]

It looks like this issue has been used for a few different problems, and I believe that they've all been addressed. I'm going to go ahead and close this issue with this PR. If I have misunderstood, please open a new issue so that we make sure each issue corresponds to one problem. Thanks.