dyninst
dyninst copied to clipboard
dyninst
dyninst is unable to parse libdw.so
The systemtap dyninst backend creates a shared object containing probe functions. The shared object cannot currently be loaded and I chased the problem down to dyninst being unable to parse libdw.so which the stap shared object references. Using the CodeDump tool located here: https://github.com/mxz297/dyninst-tools I see: ../CodeDump/CodeDump /usr/lib64/libdw-0.185.so Segmentation fault (core dumped)
If I switch to the Fedora version of libdw it works. One possibility is the rhel version is built with link time optimization. If I use the Fedora version then the systemtap dyninst backend works okay. The libdw in question is located at: https://scox.fedorapeople.org/dyninst/libdw-0.185.x86_64.so
On ppcle CodeDump works okay with libdw-0.185.ppc64le.so but fails the library load: ./mutator-ll -dynamic ./mutatee /usr/lib64/libdw-0.185.so mutator-ll: /builddir/build/BUILD/dyninst-11.0.0/dyninst-11.0.0/dataflowAPI/src/RoseInsnFactory.C:324: virtual bool Dyninst::DataflowAPI::RoseInsnPPCFactory::handleSpecialCases(entryID, SgAsmInstruction*, SgAsmOperandList*): Assertion `power_op_bclr == iapi_opcode' failed. Aborted (core dumped)
(mutator-ll.cpp is also on the fedorapeople site)
Also note that if I use a system command, e.g. echo which is also on the fedorapeople site, then the bpatch.processCreate fails. DYNINSTAPI_RT_LIB=/usr/lib64/dyninst/libdyninstAPI_RT.so LD_LIBRARY_PATH=/usr/lib64/dyninst ./mutator-ll -dynamic /usr/bin/echo /usr/lib64/libdw-0.185.so --FATAL-- #68: Dyninst was unable to create the specified process --FATAL-- #68: create process failed bootstrap Failed to create /usr/bin/echo
Above is with dyninst-11.0.0. Files on https://scox.fedorapeople.org/dyninst/ echo.ppc64le (fails processCreate) libdw-0.185.f33.x86_64.so (fails CodeDump) libdw-0.185.ppc64le.so (passes CodeDump, fails loadLibrary) libdw-0.185.x86_64.so (passes CodeDump, passes loadLibrary) mutator-ll.cpp (does a loadLibrary of a given so)
@stanfordcox I downloaded libdw-0.185.f33.x86_64.so and ran it with CodeDump compiled with latest Dyninst head. I do not see segfault. I saw that you are using Dyninst-11.0.0. I made several fixes to code parsing since 11.0.0 and these fixes are available in 11.0.1, which was just released recently. Can you retry with 11.0.1 and report which problems remain?
@mxz297 Right you are, upstream does indeed work. Should have checked that first. For a rhel release I am limited to only patching, not switching out the version. Think that will be a problem for me to pull out a patch for that? I'll build an upstream ppc64le and try it there.
@stanfordcox Patching will work. There is only one commit past v11.0.1, but it just adds a missing instruction (xsave).
@hainest Thanks for the patch. That fixes the x86_64 problem. I still see the ppc64le problem: % LD_LIBRARY_PATH=/home/scox/dyninst/install/lib DYNINSTAPI_RT_LIB=/home/scox/dyninst/install/lib/libdyninstAPI_RT.so ./mutator-ll -dynamic /usr/bin/echo /usr/lib64/libdw-0.185.so --FATAL-- #68: Dyninst was unable to create the specified process --FATAL-- #68: create process failed bootstrap Failed to create /usr/bin/echo Segmentation fault (core dumped) % LD_LIBRARY_PATH=/home/scox/dyninst/install/lib DYNINSTAPI_RT_LIB=/home/scox/dyninst/install/lib/libdyninstAPI_RT.so ./mutator-ll -dynamic ./mutatee /usr/lib64/libdw-0.185.so mutator-ll: /home/scox/dyninst/src/dataflowAPI/src/RoseInsnFactory.C:324: virtual bool Dyninst::DataflowAPI::RoseInsnPPCFactory::handleSpecialCases(entryID, SgAsmInstruction*, SgAsmOperandList*): Assertion `power_op_bclr == iapi_opcode' failed. Aborted (core dumped)
The problem is iapi_opcode is power_op_bc. I have no idea if this is right but it does get dyninst to handle libdw-0.185.so on ppc. power_op_bc is not handled; perhaps it was meant to be handled by branch_target? diff ~/dyninst/src/dataflowAPI/src/RoseInsnFactory.C~ ~/dyninst/src/dataflowAPI/src/RoseInsnFactory.C --- /home/scox/dyninst/src/dataflowAPI/src/RoseInsnFactory.C~ 2021-06-09 15:54:21.753883619 -0400 +++ /home/scox/dyninst/src/dataflowAPI/src/RoseInsnFactory.C 2021-06-23 14:17:37.854933719 -0400 @@ -319 +319 @@
- if(branch_target) {
- if(branch_target /* NEW */ || iapi_opcode == power_op_bc) {
Not sure what the issue is with loading /usr/bin/echo (or most any other /usr/bin/*) into a process
@mxz297 I'm going to have to hand this off to you. I'm afraid I've run out of knowledge.
@mxz297 @hainest BTW I narrowed down the 11.0.1 changes to find the culprit and it was this one: Skip parsing of blocks whose code buffer is null (#1033) That build of libdw must have a have an odd code block
@hainest @stanfordcox
On ppcle CodeDump works okay with libdw-0.185.ppc64le.so but fails the library load: ./mutator-ll -dynamic ./mutatee /usr/lib64/libdw-0.185.so mutator-ll: /builddir/build/BUILD/dyninst-11.0.0/dyninst-11.0.0/dataflowAPI/src/RoseInsnFactory.C:324: virtual bool Dyninst::DataflowAPI::RoseInsnPPCFactory::handleSpecialCases(entryID, SgAsmInstruction*, SgAsmOperandList*): Assertion `power_op_bclr == iapi_opcode' failed. Aborted (core dumped)
I cannot reproduce this error on a local power machine due to missing dependencies for libdw-0.185.so. It looks like loadlLibrary need to loads all its dependency.
@stanfordcox Can you post the stack trace for the assertion?
Also note that if I use a system command, e.g. echo which is also on the fedorapeople site, then the bpatch.processCreate fails. DYNINSTAPI_RT_LIB=/usr/lib64/dyninst/libdyninstAPI_RT.so LD_LIBRARY_PATH=/usr/lib64/dyninst ./mutator-ll -dynamic /usr/bin/echo /usr/lib64/libdw-0.185.so --FATAL-- #68: Dyninst was unable to create the specified process --FATAL-- #68: create process failed bootstrap Failed to create /usr/bin/echo
For this case, by setting environment variable DYNINST_DEBUG_STARTUP, we can see that dyninst cannot find main function in /usr/bin/echo. Binaries under /usr/bin are likely stripped, so there are no main symbols. I know that Dyninst has some heuristics for finding main for stripped binaries:
https://github.com/dyninst/dyninst/blob/2e78f7c5c26c8ba5c9f9388d3ba447450528dcbf/dyninstAPI/src/image.C#L480
My guess is that the heuristics failed to identify the main function for stripped binaries for latest fedora.
@mxz297 Ah you're right about echo; thought I had grabbed the debuginfo for that. It works okay with it of course.
The stapdyn testsuite seems to run okay with the RoseInsnFactory "patch" so that must at least be in the right neighborhood. Without the patch the stack trace is:
#0 __GI_raise (sig=
@mxz297 @stanfordcox Have we confirmed that this is fixed in the HEAD of master?
@hainest I believe the x86 problem is resolved in the HEAD of master. The two power problems are not, and they are two different problems.
One problem is an assertion where @stanfordcox has a patch to work around it. Based on the stack trace without the trace, it happens in jump table analysis during code parsing, jump table analysis itself has some mechanisms to tolerate certain errors, so I am fine with working around it at the moment. @stanfordcox Can you create a PR with your patch? We can review, test the patch and merge it.
The other power problem is caused by a failure in findMain to find main inside striped binaries. @stanfordcox is able to work around this problem by supplying debug file for the binary (essentially, this is still a problem). @hainest I think the strategy is to report this failure in findMain in a separate issue and move on.
@stanfordcox @mxz297 I see we have an issue for the findMain problem (#1071), but I don't see a PR for the first item. Did that ever come to fruition?