FlameGraph
FlameGraph copied to clipboard
brendangregg
Flamegraph error when passing it what seems like a valid folded file
I seem to have a valid out-perf.folded file but the flamegraph.pl program complains when run on this file: Ignored 25155 lines with invalid format ERROR: No stack counts found
The way I made the file was not the usual way. I run on an ARM target that cannot run perl. So what I do is store the contents of perf script to a file (say perf.file) and then copy said file to an x86 machine (where I can run perl).
So something like this:
#On ARM machine
perf record -F 99 -p my_pid -- sleep 20
perf script > perf_script_output_file
#On x86 machine
#Get hold of the perf_script_output_file from the ARM machine.
cat perf_script_output_file > FlameGraph/stackcollapse-perf.pl > out.perf-folded
# This runs fine without errors and when I view the out.perf-folded file I see all my stack traces.
FlameGraph/flamegraph.pl out.perf-folded > perf-kernel.svg
Ignored 25155 lines with invalid format
ERROR: No stack counts found
What should I be looking for to figure out why this is happening?
Here is a minimal program and the folded stack trace vec.txt out.perf-folded.txt
Both files are attached as .txt because github won't allow .cpp or .perf-folded extensions.
I noticed that the folded file and the output of perf_script look exactly the same (which might be the problem).
I think I see the problem:
cat perf_script_output_file.txt | FlameGraph/stackcollapse-perf.pFlameGraph/stackcollapse-perf.pl line 1: :6062: command not found
FlameGraph/stackcollapse-perf.pl: line 2: syntax error near unexpected token `('
FlameGraph/stackcollapse-perf.pl: line 2: ` ffffffc00016ba5c perf_event_exec ([kernel.kallsyms])'
It just seems like the stack collapse script is unable to parse this output from perf-script.
I did not.
On Mon, Apr 23, 2018 at 9:00 AM John Krull [email protected] wrote:
@RajivKurian https://github.com/RajivKurian did you ever find a solution to your problem?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brendangregg/FlameGraph/issues/132#issuecomment-383628423, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0hpAFw0059zMsSZ51cloN4g7hGWG8mks5trfqugaJpZM4NDaPH .
I have the exact same issue, except the output from stackcollapse-perf.pl is an empty file. Any suggestions?
Had the same problem. Tried it with ActivePerl-5.26. Now it works. Did you try to use another version of perl?
Also attaching the raw perf script file, in case it's useful perf_script_output_file.txt
Run with the data you provided, flamegraph generated, no error shown
perl stackcollapse-perf.pl perf_script_output_file.txt | perl flamegraph.pl > a.html
with
$ perl --version
This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-msys-thread-multi
Copyright 1987-2017, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
Just to let you know: I also struggled with this problem until I realized I needed to add the -g option to perf record to record stack traces, like this:
sudo perf record -g -o perf_with_stack.data my_process
sudo perf script -i perf_with_stack.data | stackcollapse-perf.pl | flamegraph.pl > flamegraph.png
using activeperl 5.28.1 fixed this perl 5.26.3 from cygwin and perl 5.22.1 from wsl did not work
My situation: Wrong: perf record -F 99 ./perf_test -g -- sleep 30 Right: perf record -F 99 -g ./perf_test -- sleep 30
Same issue on Debian Bullseye:
root@rob-prom-graf:~/git/FlameGraph# perf sched record -g -a -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.622 MB perf.data (8900 samples) ]
Stacks are recorded:
ffffffffb3e000f5 secondary_startup_64_no_verify+0xb0 ([kernel.kallsyms])
swapper 0 [003] 7260.568180: sched:sched_switch: prev_comm=swapper/3 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=perf_5.10 next_pid=5633 next_prio=120
ffffffffb46c3538 __schedule+0x5b8 ([kernel.kallsyms])
ffffffffb46c3538 __schedule+0x5b8 ([kernel.kallsyms])
ffffffffb46c3b38 schedule_idle+0x28 ([kernel.kallsyms])
ffffffffb3ec17a7 do_idle+0x167 ([kernel.kallsyms])
ffffffffb3ec1a69 cpu_startup_entry+0x19 ([kernel.kallsyms])
ffffffffb3e000f5 secondary_startup_64_no_verify+0xb0 ([kernel.kallsyms])
But, I end up with an empty folded file:
root@rob-prom-graf:~/git/FlameGraph# perf script > offcpu-deb
root@rob-prom-graf:~/git/FlameGraph# ./stackcollapse-perf-sched.awk offcpu-deb > offcpu-deb.folded
root@rob-prom-graf:~/git/FlameGraph# ls -l offcpu-deb*
-rw-r--r-- 1 root root 11577768 May 3 10:17 offcpu-deb
-rw-r--r-- 1 root root 0 May 3 10:17 offcpu-deb.folded
Perl is v5.32.1. Perf version is 5.10.113 (I first had to work around https://linux.debian.bugs.dist.narkive.com/dL9WRnXa/bug-983314-linux-base-perf-fails-to-report-that-linux-perf-5-10-is-not-installed by explicitly installing linux-perf-5.10). awk is GNU Awk 5.1.0.
Edit: adding strace file of a test run. strace_stack_collapse.txt .
The tip from https://github.com/brendangregg/FlameGraph/issues/150#issuecomment-338448925 worked on my Debian system, but not on my SLES12SP5 system (kernel 4.12.14-122.116-default).
Same issue with SLES15SP3:
hana-g8-vm-sles15sp3:~/git/FlameGraph # uname -r
5.3.18-150300.59.63-default
hana-g8-vm-sles15sp3:~/git/FlameGraph # cat /proc/sys/kernel/sched_schedstats
1
Still a 0 bytes file:
hana-g8-vm-sles15sp3:~/git/FlameGraph # ./stackcollapse-perf-sched.awk g8-vm > g8-vm.folded
hana-g8-vm-sles15sp3:~/git/FlameGraph # ls -l g8*
-rw-r--r-- 1 root root 240978 May 11 04:09 g8-vm
-rw-r--r-- 1 root root 0 May 11 04:10 g8-vm.folded
My situation: in the general user mode, using WSL2 Wrong: perf record -F 99 -g ./perf_test -- sleep 30 Right: sudo perf record -F 99 -g ./perf_test -- sleep 30
