gethostlatency no output

[oity]

gethostlatency on ubuntu Ubuntu 16.04.1 LTS (4.4.0-53-generic) after printinig colum headers produce no noticeable output

:/usr/share/bcc/tools# ./gethostlatency TIME PID COMM LATms HOST ^C

while in other terminal a test program has been ran

ltrace ./a.out www.google.it __libc_start_main(0x4006d6, 2, 0x7ffdd3ef79a8, 0x400790 <unfinished ...> gethostbyname("www.google.it") = 0x7fe3808581c0 printf("Hostname: %s\n", "www.google.it"Hostname: www.google.it ) = 24 inet_ntoa({ 0x23c63ad8 }) = "216.58.198.35" printf("IP Address: %s\n", "216.58.198.35"IP Address: 216.58.198.35 ) = 26 +++ exited (status 0) +++

[pchaigno]

I was able to reproduce the error (same OS and distrib.).

The issue seem to be that bcc doesn't attach to the "proper" C library. If I hardcode the path to my C library (the one used by curl, /lib/x86_64-linux-gnu/libc.so.6 for me), it works as expected:

diff --git a/tools/gethostlatency.py b/tools/gethostlatency.py
index 73e4ad2..eadd818 100755
--- a/tools/gethostlatency.py
+++ b/tools/gethostlatency.py
@@ -84,12 +84,13 @@ int do_return(struct pt_regs *ctx) {
 }
 """
 b = BPF(text=bpf_text)
-b.attach_uprobe(name="c", sym="getaddrinfo", fn_name="do_entry")
-b.attach_uprobe(name="c", sym="gethostbyname", fn_name="do_entry")
-b.attach_uprobe(name="c", sym="gethostbyname2", fn_name="do_entry")
-b.attach_uretprobe(name="c", sym="getaddrinfo", fn_name="do_return")
-b.attach_uretprobe(name="c", sym="gethostbyname", fn_name="do_return")
-b.attach_uretprobe(name="c", sym="gethostbyname2", fn_name="do_return")
+libc = "/lib/x86_64-linux-gnu/libc.so.6"
+b.attach_uprobe(name=libc, sym="getaddrinfo", fn_name="do_entry")
+b.attach_uprobe(name=libc, sym="gethostbyname", fn_name="do_entry")
+b.attach_uprobe(name=libc, sym="gethostbyname2", fn_name="do_entry")
+b.attach_uretprobe(name=libc, sym="getaddrinfo", fn_name="do_return")
+b.attach_uretprobe(name=libc, sym="gethostbyname", fn_name="do_return")
+b.attach_uretprobe(name=libc, sym="gethostbyname2", fn_name="do_return")

This is happening because, when resolving the path to the C library, bcc_procutils_which_so only returns the first found, even though I have 3 possible C libraries on my system:

$ ldconfig -p | grep -i "libc\."
	libc.so.6 (libc6,x32, OS ABI: Linux 3.4.0) => /libx32/libc.so.6
	libc.so.6 (libc6,x86-64, OS ABI: Linux 2.6.32) => /lib/x86_64-linux-gnu/libc.so.6
	libc.so.6 (libc6, OS ABI: Linux 2.6.32) => /lib32/libc.so.6

I'd like to submit a fix for this, but I'm not sure what's the proper fix. Should we attach to all libraries found (3 in my case)?

/cc @vmg

[goldshtn]

My kneejerk reaction is to fail the attach_uprobe call when there are multiple libraries matching the spec, but this will likely break a lot of callers using libc, pthread, and other common libraries. On the other hand, if we attach an architecture-dependent probe to a library that has a different architecture, it probably won't work as expected (as in this case).

What I suggest:

• If a pid was specified to attach_uprobe, try to use the library that's currently loaded into that process.
• If a pid wasn't specified, attach to all matching libraries and hope for the best.

Also, we might want to warn users of attach_uprobe (in the reference guide) that this is a potential pitfall.

@brendangregg @4ast Thoughts?

[vmg]

If a pid was specified to attach_uprobe, try to use the library that's currently loaded into that process.

This seems easy to implement and good UX for the end user. :+1:

If a pid wasn't specified, attach to all matching libraries and hope for the best.

Eh, this sounds like a recipe for disaster. How about attaching to the library that matches the currently running architecture for the BCC client process? It's not a perfect heuristic, but it ought to work reliably on 99% of the use cases (i.e. the version of Python running by default on the host ought to match the "default" architecture & ABI for the OS). It makes sense that if we're running BCC under a 64-bit process, we'd want to attach to libc6,x86-64.

[goldshtn]

It makes sense that if we're running BCC under a 64-bit process, we'd want to attach to libc6,x86-64.

Well, if I'm running in global mode (with no pid specified) then maybe I want to attach to as many libcs as I can 😄 Obviously some things won't work, like getting values out of registers in a platform-dependent way, but basic probing probably would? Maybe we should still allow attaching to all versions of the .so using a separate flag, but go with the arch-matching one by default?

[vmg]

Maybe we should still allow attaching to all versions of the .so using a separate flag, but go with the arch-matching one by default?

This sounds reasonable. I think the API could use an arch argument, that defaults to the arch of the currently running BCC process, and has all as a special value to attach to all possible architectures. That should cover every single user case and have a sensible default for most users.