Failed to load prog: 'Permission denied'

[zCukB]

I have installed the new package (sysmonforlinux-1.0.2-1.x86_64.rpm/sysinternalsebpf-1.0.2-1.x86_64.rpm) on RHEL 8.5 get the following error messages on startup:

Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=40 stack=0 before 2121: (0f) r7 += r6 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=40 stack=0 before 2120: (79) r7 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0_w=inv(id=0) R6_rw=invP(id=43,smin_value=-4095,smax_value=4096) R7=inv(id=0,umax_value=4096,var_off=(0x0; 0xffffffff)) R8_rw=map_value(id=0,off=4096,ks=4,vs=8192,imm=0) R9=map_valu> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: parent already had regs=40 stack=0 marks Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2131: (b7) r5 = 0 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2132: (67) r7 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2133: (77) r7 >>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2134: (25) if r7 > 0x1000 goto pc-1129 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0_w=inv(id=0,smin_value=-4095,smax_value=4095) R5_w=inv0 R6=inv(id=43,smin_value=-4095,smax_value=4096) R7_w=inv(id=0,umax_value=4096,var_off=(0x0; 0xffffffff)) R8=map_value(id=0,of> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2135: (07) r6 += -1 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2136: (67) r6 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2137: (77) r6 >>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2138: (25) if r6 > 0xffe goto pc-1133 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0=inv(id=0,smin_value=-4095,smax_value=4095) R5=inv0 R6=inv(id=0,umax_value=4094,var_off=(0x0; 0xffffffff)) R7=inv(id=0,umax_value=4096,var_off=(0x0; 0xffffffff)) R8=map_value(id=0,> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2139: (bf) r1 = r0 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2140: (67) r1 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2141: (c7) r1 s>>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2142: (b7) r2 = 1 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2143: (6d) if r2 s> r1 goto pc-1138 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0=inv(id=45,smin_value=-4095,smax_value=4095) R1_w=inv(id=0,umin_value=1,umax_value=4095,var_off=(0x0; 0xfff)) R2_w=inv1 R5=inv0 R6=inv(id=0,umax_value=4094,var_off=(0x0; 0xffffffff> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2144: (79) r1 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2145: (15) if r1 == 0x0 goto pc+5 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0=inv(id=45,smin_value=-4095,smax_value=4095) R1_w=inv(id=0,umax_value=4095,var_off=(0x0; 0xfff)) R2_w=inv1 R5=inv0 R6=inv(id=0,umax_value=4094,var_off=(0x0; 0xffffffff)) R7=inv(id=> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2146: (79) r1 = *(u64 *)(r10 -96) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2147: (79) r2 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2148: (1f) r1 -= r2 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: last_idx 2148 first_idx 2138 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=4 stack=0 before 2147: (79) r2 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2146: (79) r1 = *(u64 *)(r10 -96) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2145: (15) if r1 == 0x0 goto pc+5 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2144: (79) r1 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2143: (6d) if r2 s> r1 goto pc-1138 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2142: (b7) r2 = 1 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2141: (c7) r1 s>>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2140: (67) r1 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2139: (bf) r1 = r0 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2138: (25) if r6 > 0xffe goto pc-1133 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0_rw=inv(id=0,smin_value=-4095,smax_value=4095) R5_w=inv0 R6_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R7_w=inv(id=0,umax_value=4096,var_off=(0x0; 0xffffffff)) R8> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: parent didn't have regs=0 stack=8000 marks Apr 01 10:41:53 localhost.localdomain sysmon[10491]: last_idx 2137 first_idx 2120 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2137: (77) r6 >>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2136: (67) r6 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2135: (07) r6 += -1 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2134: (25) if r7 > 0x1000 goto pc-1129 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2133: (77) r7 >>= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2132: (67) r7 <<= 32 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2131: (b7) r5 = 0 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2130: (85) call bpf_probe_read_str#45 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2129: (bf) r3 = r8 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2128: (57) r2 &= 4095 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2127: (bf) r2 = r6 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2126: (0f) r1 += r2 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2125: (79) r1 = *(u64 *)(r10 -96) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2124: (57) r2 &= 4095 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2123: (87) r2 = -r2 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2122: (bf) r2 = r7 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2121: (0f) r7 += r6 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: regs=0 stack=8000 before 2120: (79) r7 = *(u64 *)(r10 -128) Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0_w=inv(id=0) R6_rw=invP(id=43,smin_value=-4095,smax_value=4096) R7=inv(id=0,umax_value=4096,var_off=(0x0; 0xffffffff)) R8_rw=map_value(id=0,off=4096,ks=4,vs=8192,imm=0) R9=map_valu> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: parent already had regs=0 stack=8000 marks Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2149: (b7) r2 = 47 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: 2150: (73) *(u8 *)(r1 +4095) = r2 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R0=inv(id=45,smin_value=-4095,smax_value=4095) R1_w=map_value(id=0,off=0,ks=4,vs=8192,smin_value=-4095,smax_value=0) R2_w=inv47 R5=inv0 R6=inv(id=0,umax_value=4094,var_off=(0x0; 0xff> Apr 01 10:41:53 localhost.localdomain sysmon[10491]: R1 unbounded memory access, make sure to bounds check any such access Apr 01 10:41:53 localhost.localdomain sysmon[10491]: processed 1320 insns (limit 1000000) max_states_per_insn 2 total_states 109 peak_states 109 mark_read 84 Apr 01 10:41:53 localhost.localdomain sysmon[10491]: libbpf: -- END LOG -- Apr 01 10:41:53 localhost.localdomain sysmon[10491]: libbpf: failed to load program 'sysmon/ProcCreate/rawExit' Apr 01 10:41:53 localhost.localdomain sysmon[10491]: libbpf: failed to load object './/sysmonEBPFkern4.17-5.1.o' Apr 01 10:41:53 localhost.localdomain sysmon[10491]: ERROR: failed to load prog: 'Permission denied' Apr 01 10:41:53 localhost.localdomain sysmon[10452]: Telemetry failed to start: eBPF object could not be loaded

Since I am not so deep in this topic, I post it here :-)

[MarioHewardt]

It looks like the verifier is having problems verifying the eBPF program. We'll add this to our backlog to look into.

[frkn4129]

i am facing the same situation

[git-hub-nub]

Hi Team,

Grateful for this to be resolved. We are experiencing this on our fleet of RHEL 8.5 hosts also.

[MarioHewardt]

Thanks everyone for your patience. This is the next item I will look at as soon as I get through current backlog items.

[aRustyDev]

Hey Team,

I'm working on getting this ready to deploy to our own fleet of RHEL machines. Looking forward to hearing about any resolutions/progress!

I've been trying to test this on Rocky Linux 8.6
Heres the SystemCtl output

[root@localhost ~]# systemctl status sysmon
● sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2022-07-11 16:20:34 EDT; 3s ago
  Process: 42022 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=12)

Jul 11 16:20:34 localhost.localdomain sysmon[42071]: 2149: (b7) r2 = 47
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: 2150: (73) *(u8 *)(r1 +4095) = r2
Jul 11 16:20:34 localhost.localdomain sysmon[42071]:  R0=inv(id=45,smin_value=-4095,smax_value=4095) R1_w=map_value(id=0,off=0,ks=4,vs=8192,smin_value=-4095,smax_value=0) R2_w=inv47 R5=inv0 R6=inv>
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: R1 unbounded memory access, make sure to bounds check any such access
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: processed 1320 insns (limit 1000000) max_states_per_insn 2 total_states 109 peak_states 109 mark_read 84
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: libbpf: -- END LOG --
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: libbpf: failed to load program 'sysmon/ProcCreate/rawExit'
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: libbpf: failed to load object './/sysmonEBPFkern4.17-5.1.o'
Jul 11 16:20:34 localhost.localdomain sysmon[42071]: ERROR: failed to load prog: 'Permission denied'
Jul 11 16:20:34 localhost.localdomain sysmon[42022]: Telemetry failed to start: eBPF object could not be loaded

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I installed via the RHEL specifications listed in INSTALL.md, and here

[git-hub-nub]

Thanks everyone for your patience. This is the next item I will look at as soon as I get through current backlog items.

Hi Mario, is there any ETA on a possible fix? Thanks

[MarioHewardt]

Hi @git-hub-nub - No ETA yet. As soon as I have an update, I will let everyone know.

[hsekowski]

is this and https://github.com/Sysinternals/SysmonForLinux/issues/66 duplicates?

[Lombs]

Hey guys, same issue here with our customer on RHEL 8.5. @MarioHewardt is RHEL8 part of the quality checks for SysmonForLinux? We would be very interested in this technology :)

[SirStephanikus]

1:1 same issue here on a rocky 8 system.

[timo92700]

Hello, Same issue here, on RHEL8.6 :

Still no ETA on a possible fix ?

[MarioHewardt]

Closing as I've pushed a fix that should resolve the issue. If you encounter it again, please reopen. You may have to run getOffsets (https://github.com/Sysinternals/SysinternalsEBPF/tree/main/getOffsets) in case of errors.

Please note that you will have to build Sysmon until we get new packages out.