bpftune icon indicating copy to clipboard operation
bpftune copied to clipboard

could not load skeleton: No such process

Open jakesteele opened this issue 1 year ago • 20 comments

OS: Linux - CachyOS BPFTune: bpftune-git r528.04bab5d-1

Error:

bpftune.service - BPF-based auto-tuning of system parameters
     Loaded: loaded (/usr/lib/systemd/system/bpftune.service; disabled; preset: disabled)
     Active: active (running) since Fri 2024-08-23 21:58:23 PDT; 8s ago
 Invocation: a21f610bc6a84604abf16c8936dfd910
   Main PID: 25466 (bpftune)
      Tasks: 2 (limit: 76976)
     Memory: 55.4M (peak: 65.6M)
        CPU: 1.451s
     CGroup: /system.slice/bpftune.service
             └─25466 /usr/bin/bpftune

Aug 23 21:58:23 arch systemd[1]: Started BPF-based auto-tuning of system parameters.
Aug 23 21:58:23 arch bpftune[25466]: bpftune works fully
Aug 23 21:58:23 arch bpftune[25466]: bpftune supports per-netns policy (via netns cookie)
Aug 23 21:58:24 arch bpftune[25466]: could not load skeleton: No such process

jakesteele avatar Aug 24 '24 04:08 jakesteele

can you try running bpftune standalone with debug enabled and provide output? sudo bpftune -ds will do this. thanks!

alan-maguire avatar Aug 27 '24 07:08 alan-maguire

Hey Alan, i'm using CachyOS too, here is a 10 second log after running "sudo bpftune -ds"., bpftunelog.txt

I hope it helps in diagnosing why the skeleton error appears.

EDIT: Could this be happening because CachyOS is using BBRv3 (Google) instead of Cubic? and bpftune isn't able to execute some of it's functions?

Shendisx avatar Aug 27 '24 14:08 Shendisx

From the log, the issue I see is the tuner library directory is not present:

bpftune: could not open dir '/usr/local/lib64/bpftune/': No such file or directory

As a consequence, bpftune cannot find the associated pluggable tuners.

Did you build with a non-standard INSTALLPATH by any chance?

If so, we probably need to add code to bpftune to accommodate that. We can add a -DINSTALLPATH variable or similar so the code knows where to look for tuners.

alan-maguire avatar Aug 27 '24 14:08 alan-maguire

On my case, i followed the instructions written in this repo.

Clone > cd > make ; sudo make install > sudo bpftune -ds

Compiler used: Clang 18.1.8

Shendisx avatar Aug 27 '24 15:08 Shendisx

hmm, should be there by default I would have thought. Is /usr/local/lib64/bpftune present, and does it contain .so files?

alan-maguire avatar Aug 27 '24 15:08 alan-maguire

Nope, in (Arch Linux or CachyOS) /usr/local/lib64 does not exist, only lib does

Shendisx avatar Aug 27 '24 15:08 Shendisx

So are the tuners in /usr/lib/bpftune?

alan-maguire avatar Aug 27 '24 15:08 alan-maguire

Correct, they're 8 .so files

Shendisx avatar Aug 27 '24 15:08 Shendisx

ok I'm working on some cleanup with the Makefiles/build so that things should work better for nonstandard locations. should have something to try shortly..

alan-maguire avatar Aug 27 '24 15:08 alan-maguire

Perfect! I'll make sure to try these changes when they arrive.

Shendisx avatar Aug 27 '24 15:08 Shendisx

oops I misread the log here. in your case the only issue we see is with the route table tuner; a function it relies on (fib6_age()) is gone, so that's expected. I've got some build improvements that I'll push shortly, but it looks like things are working on your setup at least. If you run bpftool prog you should see a bunch of programs attached.

alan-maguire avatar Aug 27 '24 15:08 alan-maguire

(tuners can be optionally installed in /usr/local/lib64 so we check there but if not there it's not an error)

alan-maguire avatar Aug 27 '24 15:08 alan-maguire

Indeed, bpftool prog shows a lot of them and bpftune is present too.

Shendisx avatar Aug 27 '24 16:08 Shendisx

it's possible the above-reported Aug 23 21:58:24 arch bpftune[25466]: could not load skeleton: No such process is just the route table tuner issue since bpftune reports as running. I'll try and improve logging around these sorts of failures to make that clearer.

alan-maguire avatar Aug 27 '24 16:08 alan-maguire

Ah, so the /usr/local/lib64 is just a warning i see.

Shendisx avatar Aug 27 '24 16:08 Shendisx

Yep, my bad, I forgot this. We search /usr/lib64 and /usr/local/lib64 for tuners. I'm prepping a pull request that will better support systems with /usr/lib and not /usr/lib64. Basically at build time you make libdir=lib ; sudo make install libdir=lib

alan-maguire avatar Aug 27 '24 16:08 alan-maguire

I've merged a fix that should better support /usr/lib installation+use; if you

make libdir=lib ; sudo make install libdir=lib you should get a working install in /usr/lib

As for the main issue, it requires better messaging but is not reflective of a bug, unfortunately our tracing targets come and go in Linux.

alan-maguire avatar Aug 27 '24 16:08 alan-maguire

bpftunelog.txt

Hmm is it better?

Just to be more clear maybe i wasn't (sorry), after running: make libdir=lib ; sudo make install libdir=lib

/usr/local/lib is completely empty, i think /usr/local is never used on Arch afaik (might be wrong).

And the /usr folder contains the following structure:

/usr/bin /usr/include /usr/lib /usr/lib32 /usr/lib64 (Targets lib) /usr/local /usr/sbin (Targets bin) /usr/share /usr/src

Right now it seems that bpftune still believes that the tuners are located in '/usr/local/lib/bpftune/' which they're not, instead they're in "/usr/lib/bpftune" as i told you before, is this intended?

Shendisx avatar Aug 27 '24 16:08 Shendisx

nope that's all intended; we look for tuners in /usr/lib/bpftune first, that's the standard location; we then try /usr/local/lib/bpftune also but it's fine if none are found there. without the libdir=lib we look in /usr/lib64/bpftune, /usr/local/lib64/bpftune

alan-maguire avatar Aug 27 '24 16:08 alan-maguire

Oh nice, all good then, thanks for the fix!

Shendisx avatar Aug 27 '24 16:08 Shendisx

improved error messaging with #94 so closing this one out..

alan-maguire avatar Oct 23 '24 11:10 alan-maguire