pcp
pcp copied to clipboard
performancecopilot
pmcd: unexpected end-of-file at initial exchange with bpftrace PMDA
I would like to use the bpftrace-PMDA, but I cannot seem to properly activate it. I installed pcp as the Grafana-plugin, edited /var/lib/pcp/pmdas/bpftrace/bpftrace.conf and run cd /var/lib/pcp/pmdas/bpftrace && sudo ./Install.
The output of the Install-command is:
$ cd /var/lib/pcp/pmdas/bpftrace && sudo ./Install
[sudo] password for XXX:
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bpftrace metrics have appeared ... 1 warnings, 1 metrics and 0 values
And the content of the log:
$ cat /var/log/pcp/pmcd/pmcd.log
Log for pmcd on berg.jp.net started Mon May 20 15:14:54 2024
active agent dom pid in out ver protocol parameters
============ === ===== === === === ======== ==========
root 1 1899 6 7 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot
pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f173db52a50]
proc 3 1905 9 10 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
pmproxy 4 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=pmproxy_init [0x7f173db42530]
xfs 11 1913 11 12 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 1916 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux
mmv 70 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f173db42570]
kvm 95 1920 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/kvm/pmdakvm -d 95
jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f173db39d00]
Host access list:
00 01 Cur/MaxCons host-spec host-mask lvl host-name
== == =========== ======================================= ======================================= === ==============
y y 0 0 127.0.1.1 255.255.255.255 0 localhost
y y 0 0 / / 1 unix:
n 0 0 0.0.0.0 0.0.0.0 4 .*
n 0 0 :: :: 8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off
pmcd: PID = 1894, PDU version = 2
pmcd request port(s):
sts fd port family address
=== ==== ===== ====== =======
ok 4 unix /run/pcp/pmcd.socket
ok 0 44321 inet INADDR_ANY
ok 3 44321 ipv6 INADDR_ANY
[Mon May 20 15:16:05] pmcd(1894) Info:
pmcd RESTARTED at Mon May 20 15:16:05 2024
Current PMCD clients ...
fd client connection from ipc ver operations denied
== ======================================== ======= =================
17 /run/pcp/pmcd.socket 2
18 /run/pcp/pmcd.socket 2
19 /run/pcp/pmcd.socket 2
20 /run/pcp/pmcd.socket 2
21 /run/pcp/pmcd.socket 2
22 /run/pcp/pmcd.socket 2
23 /run/pcp/pmcd.socket 2
active agent dom pid in out ver protocol parameters
============ === ===== === === === ======== ==========
root 1 1899 6 7 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot
pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f173db52a50]
proc 3 1905 9 10 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
pmproxy 4 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=pmproxy_init [0x7f173db42530]
xfs 11 1913 11 12 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 1916 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux
mmv 70 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f173db42570]
kvm 95 1920 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/kvm/pmdakvm -d 95
jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f173db39d00]
Host access list:
00 01 Cur/MaxCons host-spec host-mask lvl host-name
== == =========== ======================================= ======================================= === ==============
y y 0 0 127.0.1.1 255.255.255.255 0 localhost
y y 0 0 / / 1 unix:
n 0 0 0.0.0.0 0.0.0.0 4 .*
n 0 0 :: :: 8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off
[Mon May 20 15:16:05] pmcd(1894) Info: Reloading PMNS "DEFAULT"
[Mon May 20 15:16:06] pmcd(1894) Info:
pmcd RESTARTED at Mon May 20 15:16:06 2024
Current PMCD clients ...
fd client connection from ipc ver operations denied
== ======================================== ======= =================
17 /run/pcp/pmcd.socket 2
18 /run/pcp/pmcd.socket 2
19 /run/pcp/pmcd.socket 2
20 /run/pcp/pmcd.socket 2
21 /run/pcp/pmcd.socket 2
22 /run/pcp/pmcd.socket 2
23 /run/pcp/pmcd.socket 2
pmcd: unexpected end-of-file at initial exchange with bpftrace PMDA
active agent dom pid in out ver protocol parameters
============ === ===== === === === ======== ==========
root 1 1899 6 7 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot
pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f173db52a50]
proc 3 1905 9 10 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
pmproxy 4 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=pmproxy_init [0x7f173db42530]
xfs 11 1913 11 12 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 1916 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux
mmv 70 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f173db42570]
kvm 95 1920 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/kvm/pmdakvm -d 95
jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f173db39d00]
Host access list:
00 01 Cur/MaxCons host-spec host-mask lvl host-name
== == =========== ======================================= ======================================= === ==============
y y 0 0 127.0.1.1 255.255.255.255 0 localhost
y y 0 0 / / 1 unix:
n 0 0 0.0.0.0 0.0.0.0 4 .*
n 0 0 :: :: 8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off
[Mon May 20 15:16:06] pmcd(1894) Info: PMNS file "DEFAULT" is unchanged
[Mon May 20 15:16:56] pmcd(1894) Info:
pmcd RESTARTED at Mon May 20 15:16:56 2024
Current PMCD clients ...
fd client connection from ipc ver operations denied
== ======================================== ======= =================
17 /run/pcp/pmcd.socket 2
18 /run/pcp/pmcd.socket 2
19 /run/pcp/pmcd.socket 2
20 /run/pcp/pmcd.socket 2
21 /run/pcp/pmcd.socket 2
22 /run/pcp/pmcd.socket 2
23 /run/pcp/pmcd.socket 2
Configuration file '/etc/pcp/pmcd/pmcd.conf' unchanged
Restarting any deceased agents:
"bpftrace" agent
pmcd: unexpected end-of-file at initial exchange with bpftrace PMDA
active agent dom pid in out ver protocol parameters
============ === ===== === === === ======== ==========
root 1 1899 6 7 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot
pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f173db52a50]
proc 3 1905 9 10 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
pmproxy 4 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=pmproxy_init [0x7f173db42530]
xfs 11 1913 11 12 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 1916 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux
mmv 70 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f173db42570]
kvm 95 1920 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/kvm/pmdakvm -d 95
jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f173db39d00]
Host access list:
00 01 Cur/MaxCons host-spec host-mask lvl host-name
== == =========== ======================================= ======================================= === ==============
y y 0 0 127.0.1.1 255.255.255.255 0 localhost
y y 0 0 / / 1 unix:
n 0 0 0.0.0.0 0.0.0.0 4 .*
n 0 0 :: :: 8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off
[Mon May 20 15:16:56] pmcd(1894) Info: PMNS file "DEFAULT" is unchanged
[Mon May 20 15:17:49] pmcd(1894) Info:
pmcd RESTARTED at Mon May 20 15:17:49 2024
Current PMCD clients ...
fd client connection from ipc ver operations denied
== ======================================== ======= =================
17 /run/pcp/pmcd.socket 2
18 /run/pcp/pmcd.socket 2
19 /run/pcp/pmcd.socket 2
20 /run/pcp/pmcd.socket 2
21 /run/pcp/pmcd.socket 2
22 /run/pcp/pmcd.socket 2
23 /run/pcp/pmcd.socket 2
Configuration file '/etc/pcp/pmcd/pmcd.conf' unchanged
Restarting any deceased agents:
"bpftrace" agent
pmcd: unexpected end-of-file at initial exchange with bpftrace PMDA
active agent dom pid in out ver protocol parameters
============ === ===== === === === ======== ==========
root 1 1899 6 7 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot
pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f173db52a50]
proc 3 1905 9 10 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
pmproxy 4 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=pmproxy_init [0x7f173db42530]
xfs 11 1913 11 12 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux 60 1916 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux
mmv 70 2 dso i:7 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f173db42570]
kvm 95 1920 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/kvm/pmdakvm -d 95
jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f173db39d00]
Host access list:
00 01 Cur/MaxCons host-spec host-mask lvl host-name
== == =========== ======================================= ======================================= === ==============
y y 0 0 127.0.1.1 255.255.255.255 0 localhost
y y 0 0 / / 1 unix:
n 0 0 0.0.0.0 0.0.0.0 4 .*
n 0 0 :: :: 8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off
[Mon May 20 15:17:49] pmcd(1894) Info: PMNS file "DEFAULT" is unchanged
@jp-berg that all looks normal for an agent having some issue(s) - can you paste bpftrace.log from that same directory?
$ cat /var/log/pcp/pmcd/bpftrace.log
Log for pmdabpftrace on XXX started Mon May 20 17:46:55 2024
[Mon May 20 17:46:55] pmdabpftrace(14057) Info: manager: started pmdabpftrace process manager
[Mon May 20 17:46:55] pmdabpftrace(14057) Info: manager: using bpftrace v0.11.3
[Mon May 20 17:46:56] pmdabpftrace(14051) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt...
[Mon May 20 17:46:56] pmdabpftrace(14057) Info: script: starting script s32ba0c9248794198968db1e3d2c4d805 (name=biolatency, user=root)...
[Mon May 20 17:46:56] pmdabpftrace(14051) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/runqlat.bt...
[Mon May 20 17:46:56] pmdabpftrace(14057) Info: script: starting script sb48b6d4adbca4bd6be48ba3ed862425a (name=runqlat, user=root)...
[Mon May 20 17:46:56] pmdabpftrace(14057) Info: script: started script s32ba0c9248794198968db1e3d2c4d805 (name=biolatency, user=root, PID=14060)
[Mon May 20 17:46:56] pmdabpftrace(14057) Info: script: started script sb48b6d4adbca4bd6be48ba3ed862425a (name=runqlat, user=root, PID=14064)
[Mon May 20 17:46:56] pmdabpftrace(14057) Info: script: stopped script sb48b6d4adbca4bd6be48ba3ed862425a (name=runqlat, user=root, PID=14064) due to error: definitions.h:10:3: error: unknown type name 'pid_t'
definitions.h:22:3: error: unknown type name 'pid_t'
definitions.h:34:3: error: unknown type name 'pid_t'
definitions.h:38:3: error: unknown type name 'pid_t'
and the scripts from the logs:
$ cat /var/lib/pcp/pmdas/bpftrace/autostart/runqlat.bt
#!/usr/bin/env bpftrace
/*
* runqlat.bt CPU scheduler run queue latency as a histogram.
* For Linux, uses bpftrace, eBPF.
*
* This is a bpftrace version of the bcc tool of the same name.
*
* Copyright 2018 Netflix, Inc.
* Licensed under the Apache License, Version 2.0 (the "License")
*
* 17-Sep-2018 Brendan Gregg Created this.
* 07-Apr-2020 Andreas Gerstmayr Modified for usage with PCP bpftrace PMDA.
*/
// include: @usecs
#include <linux/sched.h>
tracepoint:sched:sched_wakeup,
tracepoint:sched:sched_wakeup_new
{
@qtime[args->pid] = nsecs;
}
tracepoint:sched:sched_switch
{
if (args->prev_state == TASK_RUNNING) {
@qtime[args->prev_pid] = nsecs;
}
$ns = @qtime[args->next_pid];
if ($ns) {
@usecs = hist((nsecs - $ns) / 1000);
}
delete(@qtime[args->next_pid]);
}
$ cat /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt
#!/usr/bin/env bpftrace
/*
* biolatency.bt Block I/O latency as a histogram.
* For Linux, uses bpftrace, eBPF.
*
* This is a bpftrace version of the bcc tool of the same name.
*
* Copyright 2018 Netflix, Inc.
* Licensed under the Apache License, Version 2.0 (the "License")
*
* 13-Sep-2018 Brendan Gregg Created this.
* 07-Apr-2020 Andreas Gerstmayr Modified for usage with PCP bpftrace PMDA.
*/
// include: @usecs
kprobe:blk_account_io_start
{
@start[arg0] = nsecs;
}
kprobe:blk_account_io_done
/@start[arg0]/
{
@usecs = hist((nsecs - @start[arg0]) / 1000);
delete(@start[arg0]);
}
due to error: definitions.h:10:3: error: unknown type name 'pid_t'
definitions.h:22:3: error: unknown type name 'pid_t'
definitions.h:34:3: error: unknown type name 'pid_t'
definitions.h:38:3: error: unknown type name 'pid_t'
Looks like bpftrace(1) cannot figure out what the pid_t type definition is for your kernel. Firstly, I'd check that this bpftrace script works from your local bpftrace installation.
$ rpm -ql bpftrace|grep runqlat.bt
/usr/share/bpftrace/tools/runqlat.bt
/usr/share/man/man8/runqlat.bt.8.gz
$ sudo /usr/share/bpftrace/tools/runqlat.bt
(the final command there should not fail - if it does, this is something to resolve there first)
Ultimately though, it sounds like something wrong with your kernel header installation, so I guess you could also try adding an include of <linux/types.h> ahead of the runqlat.bt line:
#include <linux/sched.h>