pixie
pixie copied to clipboard
Go binary version detection results in `std::bad_alloc` exception
We recently revamped how Pixie detects a binary's Go version (#1605). This aligned it with how the go version command implements its functionality as well as addressed other bugs in our previous implementation (as described in the linked PR) .
We received a report of a std::bad_alloc exception that occurs within this new logic, which causes the PEM to crash.
libc++abi: terminating due to uncaught exception of type std::bad_alloc: std::bad_alloc
E20230730 23:53:00.815058 10815 signal_action.cc:63] Caught Aborted, suspect faulting address 0x262c. Trace:
**************************
PC: @ 0x7f019b04c472 (unknown) abort
@ 0x562574f7f976 (unknown) abort_message
@ 0x562574f6981b (unknown) demangling_terminate_handler()
@ 0x562574f7f5f3 (unknown) std::__terminate()
@ 0x562574f81316 (unknown) __cxxabiv1::failed_throw()
@ 0x562574f812af (unknown) __cxa_throw
@ 0x562574f16384 (unknown) (anonymous namespace)::handle_oom()
@ 0x562574f868dd (unknown) tcmalloc::allocate_full_cpp_throw_oom()
@ 0x56257081c2ca (unknown) px::stirling::obj_tools::ElfReader::BinaryByteCode<>()
@ 0x56257081badb (unknown) px::stirling::obj_tools::ReadGoString()
@ 0x56257081cdde (unknown) px::stirling::obj_tools::ReadGoBuildVersion()
@ 0x562570749126 (unknown) px::stirling::GoTLSSymAddrs()
@ 0x56257073421f (unknown) px::stirling::UProbeManager::UpdateGoTLSSymAddrs()
@ 0x562570738cbf (unknown) px::stirling::UProbeManager::AttachGoTLSUProbes()
@ 0x56257073d5aa (unknown) px::stirling::UProbeManager::DeployGoUProbes()
@ 0x56257073ecec (unknown) px::stirling::UProbeManager::DeployUProbes()
@ 0x562570740d6b (unknown) _ZNSt3__114__thread_proxyB6v15006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN2px8stirling13UProbeManager22RunDeployUProbesThreadERKN4absl12lts_2022062313flat_hash_setINS7_2md4UPIDENSB_ @ 0x7f019b0aefd4 (unknown) (unknown)
@ 0x7f019b12f5bc (unknown) (unknown)
**************************
Until this bug is addressed, there are two (soon to be three) ways to remediate this issue:
--stirling_uprobe_opt_outwill be available in Vizier v0.14.11 (#1971) and will provide a mechanism for opting binaries out of uprobe instrumentation.- This is the preferred remediation once released.
- Deploy a Vizier v0.14.2 or earlier
- Set the
PX_STIRLING_DISABLE_GOLANG_TLS_TRACINGPEM flag to true (viapx deploy --pem_flags=PX_STIRLING_DISABLE_GOLANG_TLS_TRACING=true)
There was a recent report of this in the Pixie slack. For this problematic case, the Go application in question was built with the following configuration:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, Go BuildID=zabiqC8wfzfmrjRJ0FDq/NV0zLDU8qJwxHA8tkC-k/2e_IZ0k1YT-EJ5tgrhl9/J1BG3wmTbdYe1tMJkFv3, BuildID[sha1]=f0ea16a8de2ad5701cdea163166fc0ff33014c5e, not stripped
When I run go version -m <binary>, I get this info:
go1.20.3
path command-line-arguments
dep github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
dep golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU=
dep golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
dep golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58=
dep google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A=
dep google.golang.org/grpc v1.56.1 h1:z0dNfjIl0VpaZ9iSVjA6daGatAYwPGstTjt5vkRMFkQ=
dep google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
dep producers (devel)
dep rtp v0.0.0-00010101000000-000000000000
=> ../rtp (devel)
dep ztc_interface v0.0.0-00010101000000-000000000000
=> ./ztc_interface (devel)
build -buildmode=exe
build -compiler=gc
build CGO_ENABLED=1
build CGO_CFLAGS=
build CGO_CPPFLAGS=
build CGO_CXXFLAGS=
build CGO_LDFLAGS=
build GOARCH=amd64
build GOOS=linux
build GOAMD64=v1