pixie icon indicating copy to clipboard operation
pixie copied to clipboard

Go binary version detection results in `std::bad_alloc` exception

Open ddelnano opened this issue 2 years ago • 1 comments

We recently revamped how Pixie detects a binary's Go version (#1605). This aligned it with how the go version command implements its functionality as well as addressed other bugs in our previous implementation (as described in the linked PR) .

We received a report of a std::bad_alloc exception that occurs within this new logic, which causes the PEM to crash.

libc++abi: terminating due to uncaught exception of type std::bad_alloc: std::bad_alloc
E20230730 23:53:00.815058 10815 signal_action.cc:63] Caught Aborted, suspect faulting address 0x262c. Trace:
**************************
PC: @     0x7f019b04c472  (unknown)  abort
    @     0x562574f7f976  (unknown)  abort_message
    @     0x562574f6981b  (unknown)  demangling_terminate_handler()
    @     0x562574f7f5f3  (unknown)  std::__terminate()
    @     0x562574f81316  (unknown)  __cxxabiv1::failed_throw()
    @     0x562574f812af  (unknown)  __cxa_throw
    @     0x562574f16384  (unknown)  (anonymous namespace)::handle_oom()
    @     0x562574f868dd  (unknown)  tcmalloc::allocate_full_cpp_throw_oom()
    @     0x56257081c2ca  (unknown)  px::stirling::obj_tools::ElfReader::BinaryByteCode<>()
    @     0x56257081badb  (unknown)  px::stirling::obj_tools::ReadGoString()
    @     0x56257081cdde  (unknown)  px::stirling::obj_tools::ReadGoBuildVersion()
    @     0x562570749126  (unknown)  px::stirling::GoTLSSymAddrs()
    @     0x56257073421f  (unknown)  px::stirling::UProbeManager::UpdateGoTLSSymAddrs()
    @     0x562570738cbf  (unknown)  px::stirling::UProbeManager::AttachGoTLSUProbes()
    @     0x56257073d5aa  (unknown)  px::stirling::UProbeManager::DeployGoUProbes()
    @     0x56257073ecec  (unknown)  px::stirling::UProbeManager::DeployUProbes()
    @     0x562570740d6b  (unknown)  _ZNSt3__114__thread_proxyB6v15006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN2px8stirling13UProbeManager22RunDeployUProbesThreadERKN4absl12lts_2022062313flat_hash_setINS7_2md4UPIDENSB_    @     0x7f019b0aefd4  (unknown)  (unknown)
    @     0x7f019b12f5bc  (unknown)  (unknown)
**************************

Until this bug is addressed, there are two (soon to be three) ways to remediate this issue:

  • --stirling_uprobe_opt_out will be available in Vizier v0.14.11 (#1971) and will provide a mechanism for opting binaries out of uprobe instrumentation.
    • This is the preferred remediation once released.
  • Deploy a Vizier v0.14.2 or earlier
  • Set the PX_STIRLING_DISABLE_GOLANG_TLS_TRACING PEM flag to true (via px deploy --pem_flags=PX_STIRLING_DISABLE_GOLANG_TLS_TRACING=true)

ddelnano avatar Aug 03 '23 15:08 ddelnano

There was a recent report of this in the Pixie slack. For this problematic case, the Go application in question was built with the following configuration:

ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, Go BuildID=zabiqC8wfzfmrjRJ0FDq/NV0zLDU8qJwxHA8tkC-k/2e_IZ0k1YT-EJ5tgrhl9/J1BG3wmTbdYe1tMJkFv3, BuildID[sha1]=f0ea16a8de2ad5701cdea163166fc0ff33014c5e, not stripped

When I run go version -m <binary>, I get this info:
go1.20.3
	path	command-line-arguments
	dep	github.com/golang/protobuf	v1.5.3	h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
	dep	golang.org/x/net	v0.11.0	h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU=
	dep	golang.org/x/sys	v0.9.0	h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
	dep	golang.org/x/text	v0.10.0	h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58=
	dep	google.golang.org/genproto	v0.0.0-20230410155749-daa745c078e1	h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A=
	dep	google.golang.org/grpc	v1.56.1	h1:z0dNfjIl0VpaZ9iSVjA6daGatAYwPGstTjt5vkRMFkQ=
	dep	google.golang.org/protobuf	v1.30.0	h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
	dep	producers	(devel)	
	dep	rtp	v0.0.0-00010101000000-000000000000
	=>	../rtp	(devel)	
	
	dep	ztc_interface	v0.0.0-00010101000000-000000000000
	=>	./ztc_interface	(devel)	
	
	build	-buildmode=exe
	build	-compiler=gc
	build	CGO_ENABLED=1
	build	CGO_CFLAGS=
	build	CGO_CPPFLAGS=
	build	CGO_CXXFLAGS=
	build	CGO_LDFLAGS=
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1

ddelnano avatar Jul 23 '24 20:07 ddelnano