opentelemetry-go-instrumentation icon indicating copy to clipboard operation
opentelemetry-go-instrumentation copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

error while running instrumentors

Open vasiliy-grinko opened this issue 1 year ago • 15 comments

Describe the bug

I'm running my app with the instrumentation in K8s, I have enabled the feature gate operator.autoinstrumentation.go.

"msg":"error while running instrumentors","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: regs=200 stack=0 before 423: (63) *(u32 *)(r
10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"

Environment

  • OS: Ubuntu 22.04.3 LTS
  • Go Version: 1.19.1
  • Version: v0.3.0-alpha

Expected behavior

The instrumentation container is not crushing

Logs

{"level":"info","ts":1696862058.7040868,"caller":"cli/main.go:37","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1696862058.7041774,"caller":"opentelemetry/controller.go:119","msg":"Establishing connection to OTLP receiver ..."}
{"level":"info","ts":1696862060.7122269,"caller":"process/discover.go:55","msg":"found process","pid":1496429}
{"level":"info","ts":1696862060.7129314,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496429}
{"level":"info","ts":1696862060.7129982,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496449}
{"level":"info","ts":1696862060.7130327,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496450}
{"level":"info","ts":1696862060.7131026,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496451}
{"level":"info","ts":1696862060.7131865,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496452}
{"level":"info","ts":1696862060.713268,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496453}
{"level":"info","ts":1696862060.7133186,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496454}
{"level":"info","ts":1696862060.7219062,"caller":"process/analyze.go:94","msg":"Detaching from process","pid":1496429}
{"level":"info","ts":1696862060.7220416,"caller":"process/analyze.go:139","msg":"mmaped remote memory","start_addr":"7F9AC2800000","end_addr":"7F9AC4000000"}
{"level":"info","ts":1696862060.890518,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"net/http.(*Client).do","start":2624320,"returns":[2624852,2627125,2627214,2627635,2627830,2627927,2628224,2628322
,2628428]}
{"level":"info","ts":1696862060.890655,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"net/http.HandlerFunc.ServeHTTP","start":2933216,"returns":[2933272]}
{"level":"info","ts":1696862060.8910005,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"github.com/gorilla/mux.(*Router).ServeHTTP","start":5012320,"returns":[5012798,5013120]}
{"level":"info","ts":1696862060.891502,"caller":"cli/main.go:79","msg":"target process analysis completed","pid":1496429,"go_version":"1.19.1","dependencies":{"code.cloudfoundry.org/lager":"v2.0.0+incompatible","git.swisscom.com/ctt/ctt-integration-d
op":"(devel)","github.com/KyleBanks/depth":"v1.2.1","github.com/PuerkitoBio/purell":"v1.1.1","github.com/PuerkitoBio/urlesc":"v0.0.0-20170810143723-de5bf2ad4578","github.com/alecthomas/template":"v0.0.0-20190718012654-fb15b899a751","github.com/alecth
omas/units":"v0.0.0-20210208195552-ff826a37aa15","github.com/beorn7/perks":"v1.0.1","github.com/cespare/xxhash/v2":"v2.1.1","github.com/dgrijalva/jwt-go":"v3.2.0+incompatible","github.com/go-openapi/jsonpointer":"v0.19.5","github.com/go-openapi/jsonr
eference":"v0.19.6","github.com/go-openapi/spec":"v0.20.4","github.com/go-openapi/swag":"v0.19.15","github.com/golang/protobuf":"v1.5.2","github.com/google/uuid":"v1.3.0","github.com/gorilla/mux":"v1.8.0","github.com/josharian/intern":"v1.0.0","githu
b.com/lestrrat-go/backoff/v2":"v2.0.8","github.com/lestrrat-go/blackmagic":"v1.0.0","github.com/lestrrat-go/httpcc":"v1.0.0","github.com/lestrrat-go/iter":"v1.0.1","github.com/lestrrat-go/jwx":"v1.2.9","github.com/lestrrat-go/option":"v1.0.0","github
.com/mailru/easyjson":"v0.7.7","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/mitchellh/mapstructure":"v1.1.2","github.com/opentracing/opentracing-go":"v1.2.0","github.com/openzipkin/zipkin-go":"v0.2.6-0.20210321195400-815bc6
4e1a60","github.com/pborman/uuid":"v1.2.1","github.com/pivotal-cf/brokerapi/v8":"v8.2.0","github.com/pkg/errors":"v0.9.1","github.com/prometheus/client_golang":"v1.7.1","github.com/prometheus/client_model":"v0.2.0","github.com/prometheus/common":"v0.
21.0","github.com/prometheus/procfs":"v0.1.3","github.com/sanity-io/litter":"v1.5.0","github.com/sirupsen/logrus":"v1.8.1","github.com/swaggo/files":"v0.0.0-20190704085106-630677cd5c14","github.com/swaggo/http-swagger":"v1.0.0","github.com/swaggo/swa
g":"v1.7.0","golang.org/x/crypto":"v0.0.0-20201217014255-9d1352758620","golang.org/x/net":"v0.0.0-20211101193420-4a448f8816b3","golang.org/x/sys":"v0.0.0-20211103235746-7861aae1554b","golang.org/x/text":"v0.3.7","golang.org/x/tools":"v0.1.7","google.
golang.org/grpc":"v1.30.0","google.golang.org/protobuf":"v1.26.0","gopkg.in/alecthomas/kingpin.v2":"v2.2.6","gopkg.in/retry.v1":"v1.0.3","gopkg.in/yaml.v2":"v2.4.0"},"total_functions_found":3}
{"level":"info","ts":1696862060.8916557,"caller":"cli/main.go:85","msg":"invoking instrumentors"}
{"level":"info","ts":1696862060.89224,"logger":"allocator","caller":"allocator/allocator_linux.go:39","msg":"Loading allocator","start_addr":140302664859648,"end_addr":140302690025472}
{"level":"info","ts":1696862060.8937452,"caller":"instrumentors/runner.go:85","msg":"loading instrumentor","name":"net/http/client"}
{"level":"info","ts":1696862060.895833,"caller":"inject/injector.go:102","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"end_addr":140302690025472,"headers_ptr_pos":56,"is_registers_abi":true,"method_ptr_pos":0,"path_ptr_pos":56,"start_addr":1
40302664859648,"total_cpus":4,"url_ptr_pos":16}}
{"level":"error","ts":1696862060.9951704,"caller":"instrumentors/runner.go:88","msg":"error while loading instrumentors, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argu
ment: regs=200 stack=0 before 423: (63) *(u32 *)(r10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentors.(*Manager).load\n\t/app/internal/pkg/instrumentors/runner.go:88\ngo.ope
ntelemetry.io/auto/internal/pkg/instrumentors.(*Manager).Run\n\t/app/internal/pkg/instrumentors/runner.go:36\nmain.main\n\t/app/cli/main.go:86\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1696862060.9953918,"caller":"server/probe.go:243","msg":"closing net/http instrumentor"}
{"level":"info","ts":1696862060.9954069,"caller":"client/probe.go:226","msg":"closing net/http/client instrumentor"}
{"level":"info","ts":1696862060.9954352,"caller":"mux/probe.go:222","msg":"closing gorilla/mux instrumentor"}
{"level":"error","ts":1696862060.9954455,"caller":"cli/main.go:88","msg":"error while running instrumentors","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: regs=200 stack=0 before 423: (63) *(u32 *)(r
10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

vasiliy-grinko avatar Oct 09 '23 14:10 vasiliy-grinko

Hi, can you please retry with the latest release ? (v0.7.0-alpha) In case it is not resolved, please attach the output of uname -r

RonFed avatar Oct 16 '23 13:10 RonFed

I have this problem, how to troubleshoot go version 1.20.9 ubuntu: 20.04

{"level":"error","ts":1701165351.0630052,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:166","msg":"error while loading probes, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:166\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:121\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:150\nmain.main\n\t/app/cli/main.go:99\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
--
Tue, Nov 28 2023 5:55:51 pm | {"level":"info","ts":1701165351.4821517,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:191","msg":"Cleaning bpffs"}
Tue, Nov 28 2023 5:55:51 pm | {"level":"error","ts":1701165351.4822602,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:100","msg":"instrumentation crashed","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

zsdyx avatar Nov 28 '23 09:11 zsdyx

I have this problem, how to troubleshoot go version 1.20.9 ubuntu: 20.04

{"level":"error","ts":1701165351.0630052,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:166","msg":"error while loading probes, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:166\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:121\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:150\nmain.main\n\t/app/cli/main.go:99\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
--
Tue, Nov 28 2023 5:55:51 pm | {"level":"info","ts":1701165351.4821517,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:191","msg":"Cleaning bpffs"}
Tue, Nov 28 2023 5:55:51 pm | {"level":"error","ts":1701165351.4822602,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:100","msg":"instrumentation crashed","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Please set the environment variable OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and attach the full log.

RonFed avatar Nov 30 '23 14:11 RonFed

Hi @RonFed this is log

{"level":"info","ts":1701502016.4696648,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:91","msg":"building OpenTelemetry Go instrumentation ..."}
--
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.470296,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"error","ts":1701502018.470825,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.4708626,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:113","msg":"No symbols found in binary, trying to find functions using .gosymtab"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5066905,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.(*Client).do","Offset":2807904,"ReturnOffsets":[2808436,2810803,2810900,2811320,2811522,2811619,2811913,2812010,2812108]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067766,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.HandlerFunc.ServeHTTP","Offset":3119968,"ReturnOffsets":[3120024]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067866,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"github.com/gin-gonic/gin.(*Engine).ServeHTTP","Offset":7861312,"ReturnOffsets":[7861820]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067918,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":8282880,"ReturnOffsets":[8283196,8283240,8283269,8283310,8283427,8283569]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.506797,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":8336736,"ReturnOffsets":[8337204,8338339,8338424,8338553,8338656,8339129,8339246]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.506802,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":8376416,"ReturnOffsets":[8377291,8377425,8384119,8384769,8384945,8385046,8385281,8385617,8385751,8386465,8388136,8388382]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068076,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":8557952,"ReturnOffsets":[8558514,8558580]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068126,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":8669472,"ReturnOffsets":[8671159,8672482,8672554,8672640,8672725]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068178,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(*DB).execDC","Offset":16124320,"ReturnOffsets":[16125251,16125570,16125650,16125838,16125888]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068228,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(*DB).queryDC","Offset":16127936,"ReturnOffsets":[16129173,16129251,16129298,16129517,16129564]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5069354,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5069778,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":14}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070016,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":15}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070238,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":16}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070474,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":17}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070684,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":18}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070937,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":19}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071194,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":20}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071454,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":21}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071669,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":22}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071957,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":23}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072172,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":24}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072372,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":25}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072594,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":26}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072818,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":27}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072997,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":28}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.507321,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":29}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5073404,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":30}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.50738,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":31}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5074024,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":48}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075006,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:234","msg":"Detected linux kernel version","version":"5.4.0"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075793,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:246","msg":"mlock ret","ret":18446744073709551604}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075881,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:64","msg":"Detaching from process","pid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5076582,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:43","msg":"mmaped remote memory","start_addr":"FFFFFFFFFFFFFFF5","end_addr":"7FFF5"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5076702,"logger":"Instrumentation","caller":"app/instrumentation.go:132","msg":"target process analysis completed","pid":13,"go_version":"1.20.9","dependencies":{"-buildmode=exe":null,"-compiler=gc":null,"-ldflags=\"-s":null,"-tags=jsoniter":null,"CGO_ENABLED=0":null,"GOAMD64=v1":null,"GOARCH=amd64":null,"GOOS=linux":null,"cloud.google.com/go":"0.110.7","cloud.google.com/go/bigquery":"1.53.0","cloud.google.com/go/compute/metadata":"0.2.3","cloud.google.com/go/iam":"1.1.1","cloud.google.com/go/storage":"1.31.0","command-line-arguments":null,"github.com/ClickHouse/ch-go":"0.51.2","github.com/ClickHouse/clickhouse-go/v2":"2.5.1","github.com/Conviva-Internal/mario-compiler":"1.3.5-mai1228-1","github.com/KyleBanks/depth":"1.2.1","github.com/andybalholm/brotli":"1.0.4","github.com/apache/arrow/go/v12":"12.0.0","github.com/apache/thrift":"0.16.0","github.com/beorn7/perks":"1.0.1","github.com/cenkalti/backoff/v4":"4.2.1","github.com/cespare/xxhash/v2":"2.2.0","github.com/dablelv/go-huge-util":"0.0.48","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/dlclark/regexp2":"1.10.0","github.com/gabriel-vasile/mimetype":"1.4.2","github.com/getsentry/sentry-go":"0.19.0","github.com/gin-contrib/cors":"1.3.1","github.com/gin-contrib/sse":"0.1.0","github.com/gin-contrib/timeout":"0.0.3","github.com/gin-gonic/gin":"1.9.1","github.com/go-faster/city":"1.0.1","github.com/go-faster/errors":"0.6.1","github.com/go-logr/logr":"1.3.0","github.com/go-logr/stdr":"1.2.2","github.com/go-logr/zapr":"1.2.4","github.com/go-openapi/jsonpointer":"0.19.6","github.com/go-openapi/jsonreference":"0.20.2","github.com/go-openapi/spec":"0.20.8","github.com/go-openapi/swag":"0.22.3","github.com/go-playground/locales":"0.14.1","github.com/go-playground/universal-translator":"0.18.1","github.com/go-playground/validator/v10":"10.14.0","github.com/go-redis/redis/v8":"8.11.5","github.com/go-redsync/redsync/v4":"4.8.1","github.com/go-sql-driver/mysql":"1.7.1","github.c
  | om/goccy/go-json":"0.10.2","github.com/golang/groupcache":"0.0.0-20210331224755-41bb18bfe9da","github.com/golang/protobuf":"1.5.3","github.com/golang/snappy":"0.0.4","github.com/google/flatbuffers":"2.0.8+incompatible","github.com/google/s2a-go":"0.1.4","github.com/google/uuid":"1.3.1","github.com/googleapis/enterprise-certificate-proxy":"0.2.5","github.com/googleapis/gax-go/v2":"2.12.0","github.com/grpc-ecosystem/grpc-gateway/v2":"2.16.0","github.com/hashicorp/errwrap":"1.1.0","github.com/hashicorp/go-multierror":"1.1.1","github.com/josharian/intern":"1.0.0","github.com/json-iterator/go":"1.1.12","github.com/klauspost/compress":"1.15.15","github.com/klauspost/cpuid/v2":"2.2.4","github.com/leodido/go-urn":"1.2.4","github.com/mailru/easyjson":"0.7.7","github.com/mattn/go-isatty":"0.0.19","github.com/matttproud/golang_protobuf_extensions":"1.0.4","github.com/modern-go/concurrent":"0.0.0-20180306012644-bacd9c7ef1dd","github.com/modern-go/reflect2":"1.0.2","github.com/oliveagle/jsonpath":"0.0.0-20180606110733-2e52cf6e6852","github.com/osteele/liquid":"1.3.0","github.com/osteele/tuesday":"1.0.3","github.com/patrickmn/go-cache":"2.1.0+incompatible","github.com/paulmach/orb":"0.8.0","github.com/pelletier/go-toml/v2":"2.0.8","github.com/pierrec/lz4/v4":"4.1.17","github.com/pkg/errors":"0.9.1","github.com/prometheus/client_golang":"1.16.0","github.com/prometheus/client_model":"0.3.0","github.com/prometheus/common":"0.42.0","github.com/prometheus/procfs":"0.10.1","github.com/segmentio/asm":"1.2.0","github.com/shopspring/decimal":"1.3.1","github.com/swaggo/files":"1.0.0","github.com/swaggo/gin-swagger":"1.5.3","github.com/swaggo/swag":"1.8.10","github.com/ugorji/go/codec":"1.2.11","github.com/wangjia184/sortedset":"0.0.0-20220209072355-af6d6d227aa7","github.com/zeebo/xxh3":"1.0.2","go.opencensus.io":"0.24.0","go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin":"0.46.1","go.opentelemetry.io/otel":"1.21.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace":"1.21.0","go.opentelemetry.io/otel/exporter
  | s/otlp/otlptrace/otlptracehttp":"1.21.0","go.opentelemetry.io/otel/exporters/stdout/stdouttrace":"1.16.0","go.opentelemetry.io/otel/metric":"1.21.0","go.opentelemetry.io/otel/sdk":"1.21.0","go.opentelemetry.io/otel/trace":"1.21.0","go.opentelemetry.io/proto/otlp":"1.0.0","go.uber.org/atomic":"1.11.0","go.uber.org/multierr":"1.11.0","go.uber.org/zap":"1.24.0","golang.org/x/crypto":"0.15.0","golang.org/x/net":"0.18.0","golang.org/x/oauth2":"0.11.0","golang.org/x/sync":"0.3.0","golang.org/x/sys":"0.14.0","golang.org/x/text":"0.14.0","golang.org/x/tools":"0.9.1","golang.org/x/xerrors":"0.0.0-20220907171357-04be3eba64a2","google.golang.org/api":"0.134.0","google.golang.org/appengine":"1.6.7","google.golang.org/genproto":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/genproto/googleapis/api":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/genproto/googleapis/rpc":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/grpc":"1.59.0","google.golang.org/protobuf":"1.31.0","gopkg.in/yaml.v2":"2.4.0","gopkg.in/yaml.v3":"3.0.1","mario":null,"std":"1.20.9"},"total_functions_found":10}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5078304,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:98","msg":"starting instrumentation..."}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5081096,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:178","msg":"Mounting bpffs","allocations_details":{"StartAddr":18446744073709551605,"EndAddr":524277}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5084653,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.6622827,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc/server"}
Sat, Dec 2 2023 3:26:58 pm | Verifier log: load program: no space left on device:
Sat, Dec 2 2023 3:26:58 pm | func#0 @0
Sat, Dec 2 2023 3:26:58 pm | 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; if (is_registers_abi)
Sat, Dec 2 2023 3:26:58 pm | 0: (18) r2 = 0xffff8c5811f750f8
Sat, Dec 2 2023 3:26:58 pm | 2: R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 2: (71) r2 = *(u8 *)(r2 +0)
Sat, Dec 2 2023 3:26:58 pm | R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 3: R1=ctx(id=0,off=0,imm=0) R2_w=inv1 R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; if (is_registers_abi)
Sat, Dec 2 2023 3:26:58 pm | 3: (15) if r2 == 0x0 goto pc+2
Sat, Dec 2 2023 3:26:58 pm | last_idx 3 first_idx 0
Sat, Dec 2 2023 3:26:58 pm | regs=4 stack=0 before 2: (71) r2 = *(u8 *)(r2 +0)
Sat, Dec 2 2023 3:26:58 pm | 4: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; return (void *)GO_PARAM2(ctx);
Sat, Dec 2 2023 3:26:58 pm | 4: (79) r7 = *(u64 *)(r1 +40)
Sat, Dec 2 2023 3:26:58 pm | 5: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 5: (05) goto pc+9
Sat, Dec 2 2023 3:26:58 pm | 15: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
Sat, Dec 2 2023 3:26:58 pm | 15: (18) r1 = 0xffff8c5811f750e8
Sat, Dec 2 2023 3:26:58 pm | 17: R1_w=map_value(id=0,off=24,ks=4,vs=64,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 17: (79) r1 = *(u64 *)(r1 +0)
Sat, Dec 2 2023 3:26:58 pm | R1_w=map_value(id=0,off=24,ks=4,vs=64,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 18: R1_w=inv8 R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
Sat, Dec 2 2023 3:26:58 pm | 18: (bf) r3 = r7
Sat, Dec 2 2023 3:26:58 pm | 19: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 19: (0f) r3 += r1
Sat, Dec 2 2023 3:26:58 pm | 20: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 20: (b7) r6 = 0
Sat, Dec 2 2023 3:26:58 pm | 21: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=inv0 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; struct go_slice header_fields = {};
Sat, Dec 2 2023 3:26:58 pm | 21: (7b) *(u64 *)(r10 -8) = r6
Sat, Dec 2 2023 3:26:58 pm | last_idx 21 first_idx 0
Sat, Dec 2 2023 3:26:58 pm | regs=40 stack=0 before 20: (b7) r6 = 0
Sat, Dec 2 2023 3:26:58 pm | 22: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 22: (7b) *(u64 *)(r10 -16) = r6
Sat, Dec 2 2023 3:26:58 pm | 23: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 23: (7b) *(u64 *)(r10 -24) = r6
Sat, Dec 2 2023 3:26:58 pm | 24: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 24: (bf) r1 = r10
Sat, Dec 2 2023 3:26:58 pm | 25: R1_w=fp0 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | ;
Sat, Dec 2 2023 3:26:58 pm | 25: (07) r1 += -24
Sat, Dec 2 2023 3:26:58 pm | 26: R1_w=fp-24 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));

zsdyx avatar Dec 02 '23 07:12 zsdyx

Is this error because my go.mod file version cannot be parsed，Can you tell me which one

func parseModules(mod string, logger logr.Logger) map[string]*version.Version {
	lines := strings.Split(mod, "\n")
	result := make(map[string]*version.Version)
	for _, line := range lines {
		parts := strings.Fields(line)
		if len(parts) > 1 {
			modType := parts[0]
			modPackage := parts[1]
			result[modPackage] = nil
			if modType == "dep" {
				if len(parts) < 3 {
					logger.Error(fmt.Errorf("missing version: %s", modPackage), "error parsing module info")
					continue
				}
				v, err := version.NewVersion(parts[2])
				if err != nil {
					logger.Error(err, "error parsing module info")
					continue
				}
				result[modPackage] = v
			}
		}
	}

	return result
}

zsdyx avatar Dec 02 '23 08:12 zsdyx

Why won't this packet parse {"level":"info","ts":1701682620.3500223,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"version detected---->","version":"v0.0.0-20230706204954-ccb25ca9f130","package":"google.golang.org/genproto","type":"dep"}

zsdyx avatar Dec 04 '23 09:12 zsdyx

Hi @zsdyx I don't think the error you get is a result of the parsing. I updated the way we print the verifier log to print its tail. Can you please re-run this with the latest main branch and with OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and attach the log ? (I can't reproduce this issue on my side)

RonFed avatar Dec 09 '23 08:12 RonFed

Hi @RonFed I also have similar issue

Describe the bug

I'm running my app with the instrumentation in EKS, I have enabled the feature gate operator.autoinstrumentation.go.

{"level":"error","ts":1704652636.7818418,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Environment

  • OS: Amazon linux 2
  • Go Version: 1.18
  • Version: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha

To Reproduce

Steps to reproduce the behavior: Instrumentation config: go: env: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: >- http://simplest-collector.opentelemetry-operator-system.svc.cluster.local:4318 - name: OTEL_TRACES_SAMPLER value: always_on - name: OTEL_RESOURCE_ATTRIBUTES value: env=dev - name: OTEL_METRICS_EXPORTER value: prometheus - name: OTEL_GO_AUTO_SHOW_VERIFIER_LOG value: 'true' image: >- ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha resourceRequirements: limits: cpu: 500m requests: cpu: 50m memory: 32Mi

Deployment annotations: annotations: instrumentation.opentelemetry.io/inject-go: 'true' instrumentation.opentelemetry.io/otel-go-auto-target-exe: /main

opentelemetry-auto-instrumentation container restarting with exit 0 and with below error.

{"level":"error","ts":1704652636.7818418,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
....
...
...
{"level":"info","ts":1704652637.253412,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc/server"}
Verifier log: load program: no space left on device:
	func#0 @0
	0: R1=ctx(id=0,off=0,imm=0) R10=fp0
	; if (is_registers_abi)
	0: (18) r2 = 0xffff888473a92af8
	2: R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
	2: (71) r2 = *(u8 *)(r2 +0)
	 R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
	3: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	; if (is_registers_abi)
	3: (15) if r2 == 0x0 goto pc+2
	 R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	4: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	; return (void *)GO_PARAM2(ctx);
	4: (79) r7 = *(u64 *)(r1 +40)
	5: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R7_w=inv(id=0) R10=fp0
	5: (05) goto pc+9
	15: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R7_w=inv(id=0) R10=fp0
	; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
	15: (18) r1 = 0xffff888473a92ae8

Container details:

/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 100.0G     49.8G     50.2G  50% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    30.9G         0     30.9G   0% /sys/fs/cgroup
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /configMap
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/hosts
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /dev/termination-log
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/hostname
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/resolv.conf
tmpfs                    58.7G      4.0K     58.7G   0% /run/secrets/eks.amazonaws.com/serviceaccount
tmpfs                    58.7G     12.0K     58.7G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    30.9G         0     30.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    30.9G         0     30.9G   0% /sys/firmware
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    7 root      8:18 ./main
  186 root      0:00 sh -c clear; (bash || ash || sh)
  193 root      0:00 ash
  256 root      0:00 sh -c clear; (bash || ash || sh)
  263 root      0:00 ash
  266 root      0:00 ps aux
/ # 

/ # uname -a
Linux <app-name>-588b7bfb48-l4qzn 5.4.249-163.359.amzn2.x86_64 #1 SMP Wed Jul 12 18:58:58 UTC 2023 x86_64 Linux

vbavireddi avatar Jan 07 '24 18:01 vbavireddi

@vbavireddi please try with the latest version https://github.com/open-telemetry/opentelemetry-go-instrumentation/releases/tag/v0.10.0-alpha There were few PRs regarding the gRPC server instrumentation which seems to be the problem in the log you attached.

RonFed avatar Jan 07 '24 19:01 RonFed

@RonFed Thanks for the update. I've updated the version. Still the container restarting continuously.

image: >- ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.10.0-alpha

Error log:

{"level":"error","ts":1704658707.8408637,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} ... ... {"level":"info","ts":1704658707.8676953,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin"} {"level":"info","ts":1704658707.8677032,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."} {"level":"info","ts":1704658707.8677793,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:177","msg":"Mounting bpffs","allocations_details":{"StartAddr":139637300117504,"EndAddr":139637301166080}} {"level":"info","ts":1704658707.8678958,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"google.golang.org/grpc/client"} {"level":"info","ts":1704658708.169515,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"google.golang.org/grpc/server"} Verifier log: load program: no space left on device: (14227 line(s) omitted) {"level":"error","ts":1704658708.521125,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:165","msg":"error while loading probes, cleaning up","name":"google.golang.org/grpc/server","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 54: (67) r3 <<= 32: 55: R0=inv(id=0) R1_w=fp-122 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320, (truncated, 14325 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:165\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:120\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:152\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} 49: (71) r3 = *(u8 *)(r1 +0) 50: R0=inv(id=0) R1_w=fp-124 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smax_value=9223372036854775792,umax_value=18446744073709551600,var_off=(0x0; 0xfffffffffffffff0)) R6_w=invP1 R7=inv(id=0) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=0000000m fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=mmmmmmmm fp-384=mmmmmmmm fp-392=mmmmmmmm fp-400=mmmmmmmm fp-408=mmmmmmmm fp-416=mmmmmmmm fp-424=fp fp-432=fp 50: (67) r3 <<= 56 51: R0=inv(id=0) R1_w=fp-124 R2_w=inv(id=0) R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000)) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smax_value=9223372036854775792,umax_value=18446744073709551600,var_off=(0x0; 0xfffffffffffffff0)) R6_w=invP1 R7=inv(id=0) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=0000000m fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=mmmmmmmm fp-384=mmmmmmmm fp-392=mmmmmmmm fp-400=mmmmmmmm fp-408=mmmmmmmm fp-416=mmmmmmmm fp-424=fp fp-432=fp 51: (c7) r3 s>>= 56

vbavireddi avatar Jan 07 '24 20:01 vbavireddi

@RonFed for restapi ("name":"net/http/server") also we are getting similar error.

Version: /app # uname -a Linux <cutom-image> 5.4.249-163.359.amzn2.x86_64 #1 SMP Wed Jul 12 18:58:58 UTC 2023 x86_64 Linux Base img: golang:1.20.5-alpine3.18 ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.10.0-alpha

Error: `{"level":"info","ts":1704698393.3934438,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false} {"level":"info","ts":1704698395.394087,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":7} {"level":"error","ts":1704698395.394404,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} {"level":"info","ts":1704698395.3945549,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:113","msg":"No symbols found in binary, trying to find functions using .gosymtab"} {"level":"info","ts":1704698395.4090765,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.serverHandler.ServeHTTP","Offset":4352192,"ReturnOffsets":[4353032,4353076]}} {"level":"info","ts":1704698395.4091427,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.(*Transport).roundTrip","Offset":4404448,"ReturnOffsets":[4404956,4405034,4405520,4405757,4405851,4406026,4406044,4406790,4406877,4406945,4407040,4407118,4407186,4407612,4407828]}} {"level":"info","ts":1704698395.4091587,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":4749184,"ReturnOffsets":[4749500,4749544,4749573,4749614,4749731,4749873]}} {"level":"info","ts":1704698395.409165,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":4804128,"ReturnOffsets":[4804596,4805751,4805837,4805959,4806061,4806551,4806669]}} {"level":"info","ts":1704698395.409169,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":4844608,"ReturnOffsets":[4845501,4845635,4852641,4853281,4853457,4853558,4853793,4854129,4854263,4854977,4856648,4856894]}} {"level":"info","ts":1704698395.409186,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":5251584,"ReturnOffsets":[5252146,5252212]}} {"level":"info","ts":1704698395.4091961,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":5358112,"ReturnOffsets":[5359252,5360544,5360602,5360672,5360741]}} {"level":"info","ts":1704698395.4092786,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":13} {"level":"info","ts":1704698395.4093184,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":14} {"level":"info","ts":1704698395.4093506,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":15} {"level":"info","ts":1704698395.409393,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":16} {"level":"info","ts":1704698395.4094226,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":17} {"level":"info","ts":1704698395.4094503,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":18} {"level":"info","ts":1704698395.409479,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":19} {"level":"info","ts":1704698395.4095564,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":20} {"level":"info","ts":1704698395.4095926,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":21} {"level":"info","ts":1704698395.4096258,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":22} {"level":"info","ts":1704698395.4096704,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":23} {"level":"info","ts":1704698395.409714,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":24} {"level":"info","ts":1704698395.4097433,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":44} {"level":"info","ts":1704698395.409792,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":45} {"level":"info","ts":1704698395.4098186,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":7} {"level":"info","ts":1704698395.4102046,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:234","msg":"Detected linux kernel version","version":"5.4.0"} {"level":"info","ts":1704698395.4103284,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:246","msg":"mlock ret","ret":0} {"level":"info","ts":1704698395.4103427,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:64","msg":"Detaching from process","pid":7} {"level":"info","ts":1704698395.410408,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:43","msg":"mmaped remote memory","start_addr":"7F7CB0D73000","end_addr":"7F7CB0E73000"} {"level":"info","ts":1704698395.4104197,"logger":"Instrumentation","caller":"app/instrumentation.go:134","msg":"target process analysis completed","pid":7,"go_version":"1.20.5","dependencies":{"-buildmode=exe":null,"-compiler=gc":null,"-ldflags="-s":null,"-tags=musl":null,"CGO_CFLAGS=":null,"CGO_CPPFLAGS=":null,"CGO_CXXFLAGS=":null,"CGO_ENABLED=1":null,"CGO_LDFLAGS=":null,"GOAMD64=v1":null,"GOARCH=amd64":null,"GOOS=linux":null,"command-line-arguments":null,"github.com/ZscalerCWP/cwp-gloo-custom-auth-server":null,"github.com/ZscalerCWP/zpc-api-key-management":"1.0.1","github.com/ZscalerCWP/zpc-common/cache":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/http/middlewares/error":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/http/writer":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/prometheus":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/tracing":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/viper":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/zerolog":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-framework":"1.0.6","github.com/alecthomas/kong":"0.7.1","github.com/auth0/go-jwt-middleware/v2":"2.1.0","github.com/beorn7/perks":"1.0.1","github.com/cenkalti/backoff/v4":"4.2.0","github.com/cespare/xxhash/v2":"2.2.0","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/fsnotify/fsnotify":"1.6.0","github.com/go-logr/logr":"1.2.3","github.com/go-logr/stdr":"1.2.2","github.com/golang/protobuf":"1.5.3","github.com/google/uuid":"1.3.0","github.com/grpc-ecosystem/grpc-gateway/v2":"2.16.0","github.com/hashicorp/hcl":"1.0.0","github.com/labstack/echo/v4":"4.10.2","github.com/labstack/gommon":"0.4.0","github.com/magiconair/properties":"1.8.7","github.com/mattn/go-colorable":"0.1.13","github.com/mattn/go-isatty":"0.0.17","github.com/matttproud/golang_protobuf_extensions":"1.0.4","github.com/mitchellh/mapstructure":"1.5.0","github.com/openzipkin/zipkin-go":"0.4.1","github.com/pelletier/go-toml/v2":"2.0.7","github.com/pkg/errors":"0.9.1","github.com/prometheus/client_golang":"1.14.0","github.com/prometheus/client_model":"0.3.0","github.com/prometheus/common":"0.41.0","github.com/prometheus/procfs":"0.9.0","github.com/redis/go-redis/extra/rediscmd/v9":"9.0.2","github.com/redis/go-redis/v9":"9.0.2","github.com/rs/zerolog":"1.29.0","github.com/spf13/afero":"1.9.5","github.com/spf13/cast":"1.5.1","github.com/spf13/jwalterweatherman":"1.1.0","github.com/spf13/pflag":"1.0.5","github.com/spf13/viper":"1.15.0","github.com/subosito/gotenv":"1.4.2","github.com/valyala/bytebufferpool":"1.0.0","github.com/valyala/fasttemplate":"1.2.2","go.opentelemetry.io/otel":"1.14.0","go.opentelemetry.io/otel/exporters/jaeger":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/internal/retry":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc":"1.14.0","go.opentelemetry.io/otel/exporters/stdout/stdouttrace":"1.14.0","go.opentelemetry.io/otel/exporters/zipkin":"1.14.0","go.opentelemetry.io/otel/sdk":"1.14.0","go.opentelemetry.io/otel/trace":"1.14.0","go.opentelemetry.io/proto/otlp":"0.19.0","go.uber.org/atomic":"1.11.0","go.uber.org/dig":"1.16.1","go.uber.org/fx":"1.19.3","go.uber.org/multierr":"1.11.0","go.uber.org/zap":"1.24.0","golang.org/x/crypto":"0.9.0","golang.org/x/exp":"0.0.0-20230522175609-2e198f4a06a1","golang.org/x/net":"0.10.0","golang.org/x/sys":"0.8.0","golang.org/x/text":"0.9.0","google.golang.org/genproto/googleapis/api":"0.0.0-20230530153820-e85fd2cbaebc","google.golang.org/genproto/googleapis/rpc":"0.0.0-20230530153820-e85fd2cbaebc","google.golang.org/grpc":"1.55.0","google.golang.org/protobuf":"1.30.0","gopkg.in/go-jose/go-jose.v2":"2.6.1","gopkg.in/ini.v1":"1.67.0","gopkg.in/square/go-jose.v2":"2.6.0","gopkg.in/yaml.v3":"3.0.1","std":"1.20.5"},"total_functions_found":7} {"level":"info","ts":1704698395.4105577,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin"} {"level":"info","ts":1704698395.410568,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"database/sql"} {"level":"info","ts":1704698395.410572,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."} {"level":"info","ts":1704698395.4106522,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:177","msg":"Mounting bpffs","allocations_details":{"StartAddr":140173519564800,"EndAddr":140173520613376}} {"level":"info","ts":1704698395.4108272,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"net/http/server"} {"level":"error","ts":1704698395.7103634,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:165","msg":"error while loading probes, cleaning up","name":"net/http/server","error":"field UprobeHandlerFuncServeHTTP: program uprobe_HandlerFunc_ServeHTTP: load program: no space left on device: 496: (71) r1 = *(u8 *)(r10 -45): 497: R0=in (truncated, 32578 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:165\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:120\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:152\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} {"level":"info","ts":1704698395.710505,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:190","msg":"Cleaning bpffs"} {"level":"error","ts":1704698395.710586,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"field UprobeHandlerFuncServeHTTP: program uprobe_HandlerFunc_ServeHTTP: load program: no space left on device: 496: (71) r1 = *(u8 *)(r10 -45): 497: R0=in (truncated, 32578 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} Verifier log: load program: no space left on device: (32480 line(s) omitted) propagating r8 propagating fp-176

from 497 to 467: safe
propagating r8
propagating fp-176

from 495 to 467: safe
propagating r8
propagating fp-176

from 493 to 467: safe
propagating r8
propagating fp-176

from 492 to 515: safe
propagating r8
propagating fp-176

from 485 to 468: safe
propagating r8
propagating fp-176

from 483 to 468: safe
propagating r4
last_idx 478 first_idx 535
regs=10 stack=0 before 541: (6d) if r4 s> r0 goto pc-64
regs=11 stack=0 before 540: (b7) r4 = 0
regs=1 stack=0 before 539: (85) call bpf_probe_read#4

from 541 to 478: safe
467: R0=inv(id=0) R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=map_value(id=0,off=0,ks=4,vs=736,imm=0) R7=map_value(id=0,off=208,ks=4,vs=336,imm=0) R8=invP3 R9=map_value(id=0,off=64,ks=4,vs=336,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=m??????? fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmm? fp-64=mmmmmmmm fp-72=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmm???? fp-160=map_value fp-168=mmmmmmmm fp-176=invP3 fp-184=mmmmmmmm fp-192=map_value fp-200=map_value
;
467: (79) r2 = *(u64 *)(r10 -160)
468: R0=inv(id=0) R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=map_value(id=0,off=0,ks=4,vs=336,imm=0) R6=map_value(id=0,off=0,ks=4,vs=736,imm=0) R7=map_value(id=0,off=208,ks=4,vs=336,imm=0) R8=invP3 R9=map_value(id=0,off=64,ks=4,vs=336,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=m??????? fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmm? fp-64=mmmmmmmm fp-72=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmm???? fp-160=map_value fp-168=mmmmmmmm fp-176=invP3 fp-184=mmmmmmmm fp-192=map_value fp-200=map_value
; for (u64 i = 0; i < 8; i++)`

@vasiliy-grinko Can you please share if you find any workaround for this issue.

vbavireddi avatar Jan 08 '24 07:01 vbavireddi

Hi! Any updates on this issue? I tested the latest version (v0.11.0-alpha) and got a similar error:

{"level":"info","ts":1712064476.9977193,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1712064479.021917,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:77","msg":"found process","pid":13}
{"level":"error","ts":1712064479.0235422,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"error parsing module version","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:98\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":1712064479.2411113,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"net/http.serverHandler.ServeHTTP","Offset":3181888,"ReturnOffsets":[3182728,3182772]}}
{"level":"info","ts":1712064479.2412572,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"net/http.(*Transport).roundTrip","Offset":3234144,"ReturnOffsets":[3234652,3234730,3235216,3235453,3235547,3235722,3235740,3236486,3236573,3236641,3236736,3236814,3236882,3237308,3237524]}}
{"level":"info","ts":1712064479.2412677,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":7689856,"ReturnOffsets":[7690172,7690216,7690245,7690286,7690451,7690791]}}
{"level":"info","ts":1712064479.2412744,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":7740896,"ReturnOffsets":[7741268,7742406,7742491,7742617,7742720,7743193,7743221]}}
{"level":"info","ts":1712064479.2412806,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":7780576,"ReturnOffsets":[7781436,7781555,7787818,7788481,7788642,7788728,7788848,7789584,7791129,7791569]}}
{"level":"info","ts":1712064479.2412875,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":8194752,"ReturnOffsets":[8195314,8195380]}}
{"level":"info","ts":1712064479.241293,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":8298528,"ReturnOffsets":[8299668,8300960,8301018,8301088,8301157]}}
{"level":"info","ts":1712064479.2414215,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:51","msg":"Requesting memory allocation","size":262144,"page size":4096,"cpu count":8}
{"level":"info","ts":1712064479.2416844,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":13}
{"level":"info","ts":1712064479.2417872,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":14}
{"level":"info","ts":1712064479.3204803,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":15}
{"level":"info","ts":1712064479.3207407,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":16}
{"level":"info","ts":1712064479.3207974,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":17}
{"level":"info","ts":1712064479.3208456,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":18}
{"level":"info","ts":1712064479.3209286,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":19}
{"level":"info","ts":1712064479.3211067,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":20}
{"level":"info","ts":1712064479.3211799,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":21}
{"level":"info","ts":1712064479.321278,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":22}
{"level":"info","ts":1712064479.3213801,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":23}
{"level":"info","ts":1712064479.3214848,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":24}
{"level":"info","ts":1712064479.3215709,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":25}
{"level":"info","ts":1712064479.321801,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:94","msg":"Set memlock on process successfully"}
{"level":"info","ts":1712064479.3221786,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:250","msg":"Detected linux kernel version","version":"5.15.0"}
{"level":"info","ts":1712064479.322465,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:262","msg":"mlock ret","ret":0}
{"level":"info","ts":1712064479.322523,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:84","msg":"Detaching from process","pid":13}
{"level":"info","ts":1712064479.3226507,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:62","msg":"mmaped remote memory","start_addr":"0x7fcae8da0000","end_addr":"0x7fcae8de0000"}
{"level":"info","ts":1712064479.322679,"logger":"Instrumentation","caller":"app/instrumentation.go:133","msg":"target process analysis completed","pid":13,"go_version":"1.20.14","dependencies":{"cloud.google.com/go":"0.97.0","github.com/Azure/azure-sdk-for-go":"32.4.0+incompatible","github.com/Azure/go-autorest/autorest":"0.11.27","github.com/Azure/go-autorest/autorest/adal":"0.9.20","github.com/Azure/go-autorest/autorest/azure/auth":"0.5.11","github.com/Azure/go-autorest/autorest/azure/cli":"0.4.5","github.com/Azure/go-autorest/autorest/date":"0.3.0","github.com/Azure/go-autorest/autorest/to":"0.4.0","github.com/Azure/go-autorest/autorest/validation":"0.3.1","github.com/Azure/go-autorest/logger":"0.2.1","github.com/Azure/go-autorest/tracing":"0.6.0","github.com/OpenDNS/vegadns2client":"0.0.0-20180418235048-a3fa4a771d87","github.com/PuerkitoBio/purell":"1.1.1","github.com/PuerkitoBio/urlesc":"0.0.0-20170810143723-de5bf2ad4578","github.com/akamai/AkamaiOPEN-edgegrid-golang":"1.2.1","github.com/aliyun/alibaba-cloud-sdk-go":"1.61.1755","github.com/andres-erbsen/clock":"0.0.0-20160526145045-9e14626cd129","github.com/armon/go-metrics":"0.3.9","github.com/armon/go-radix":"1.0.0","github.com/aws/aws-sdk-go":"1.39.0","github.com/beorn7/perks":"1.0.1","github.com/boombuler/barcode":"1.0.1-0.20190219062509-6c824513bacc","github.com/cenkalti/backoff/v3":"3.0.0","github.com/cenkalti/backoff/v4":"4.1.3","github.com/census-instrumentation/opencensus-proto":"0.3.0","github.com/cespare/xxhash/v2":"2.1.2","github.com/civo/civogo":"0.3.11","github.com/cloudflare/cloudflare-go":"0.49.0","github.com/cncf/xds/go":"0.0.0-20220314180256-7f1daf1720fc","github.com/cpu/goacmedns":"0.1.1","github.com/davecgh/go-spew":"1.1.1","github.com/deepmap/oapi-codegen":"1.9.1","github.com/dimchansky/utfbom":"1.1.1","github.com/dnsimple/dnsimple-go":"0.71.1","github.com/emicklei/go-restful/v3":"3.8.0","github.com/envoyproxy/go-control-plane":"0.10.3","github.com/envoyproxy/protoc-gen-validate":"0.6.7","github.com/evanphx/json-patch/v5":"5.6.0","github.com/exoscale/egoscale":"0.90.0","github.com/fatih/color":"1.13.0","github.com/fatih/structs":"1.1.0","github.com/fsnotify/fsnotify":"1.5.4","github.com/ghodss/yaml":"1.0.0","github.com/go-acme/lego/v4":"4.9.0","github.com/go-errors/errors":"1.0.1","github.com/go-logr/logr":"1.2.3","github.com/go-logr/zapr":"1.2.3","github.com/go-openapi/jsonpointer":"0.19.5","github.com/go-openapi/jsonreference":"0.19.5","github.com/go-openapi/swag":"0.19.14","github.com/go-resty/resty/v2":"2.1.1-0.20191201195748-d7b97669fe48","github.com/gogo/protobuf":"1.3.2","github.com/golang-jwt/jwt/v4":"4.2.0","github.com/golang/groupcache":"0.0.0-20210331224755-41bb18bfe9da","github.com/golang/protobuf":"1.5.2","github.com/golang/snappy":"0.0.4","github.com/google/gnostic":"0.5.7-v3refs","github.com/google/go-cmp":"0.5.8","github.com/google/go-querystring":"1.1.0","github.com/google/gofuzz":"1.1.0","github.com/google/uuid":"1.3.0","github.com/googleapis/gax-go/v2":"2.1.0","github.com/gophercloud/gophercloud":"1.0.0","github.com/gophercloud/utils":"0.0.0-20210216074907-f6de111f2eae","github.com/hashicorp/errwrap":"1.1.0","github.com/hashicorp/go-cleanhttp":"0.5.2","github.com/hashicorp/go-hclog":"1.2.0","github.com/hashicorp/go-immutable-radix":"1.3.1","github.com/hashicorp/go-multierror":"1.1.1","github.com/hashicorp/go-plugin":"1.4.3","github.com/hashicorp/go-retryablehttp":"0.7.1","github.com/hashicorp/go-rootcerts":"1.0.2","github.com/hashicorp/go-secure-stdlib/mlock":"0.1.1","github.com/hashicorp/go-secure-stdlib/parseutil":"0.1.6","github.com/hashicorp/go-secure-stdlib/strutil":"0.1.2","github.com/hashicorp/go-sockaddr":"1.0.2","github.com/hashicorp/go-uuid":"1.0.2","github.com/hashicorp/go-version":"1.6.0","github.com/hashicorp/golang-lru":"0.5.4","github.com/hashicorp/hcl":"1.0.0","github.com/hashicorp/vault/api":"1.8.0","github.com/hashicorp/vault/sdk":"0.6.0","github.com/hashicorp/yamux":"0.0.0-20180604194846-3520598351bb","github.com/iij/doapi":"0.0.0-20190504054126-0bbf12d6d7df","github.com/imdario/mergo":"0.3.12","github.com/infobloxopen/infoblox-go-client":"1.1.1","github.com/jmespath/go-jmespath":"0.4.0","github.com/josharian/intern":"1.0.0","github.com/json-iterator/go":"1.1.12","github.com/k0kubun/go-ansi":"0.0.0-20180517002512-3bf9e2903213","github.com/kolo/xmlrpc":"0.0.0-20200310150728-e0350524596b","github.com/labbsr0x/bindman-dns-webhook":"1.0.2","github.com/labbsr0x/goh":"1.0.1","github.com/linode/linodego":"1.9.1","github.com/liquidweb/go-lwApi":"0.0.5","github.com/liquidweb/liquidweb-cli":"0.6.9","github.com/liquidweb/liquidweb-go":"1.6.3","github.com/mailru/easyjson":"0.7.6","github.com/mattn/go-colorable":"0.1.12","github.com/mattn/go-isatty":"0.0.16","github.com/matttproud/golang_protobuf_extensions":"1.0.2-0.20181231171920-c182affec369","github.com/miekg/dns":"1.1.50","github.com/mimuret/golang-iij-dpf":"0.7.1","github.com/mitchellh/copystructure":"1.0.0","github.com/mitchellh/go-homedir":"1.1.0","github.com/mitchellh/go-testing-interface":"1.0.0","github.com/mitchellh/mapstructure":"1.5.0","github.com/mitchellh/reflectwalk":"1.0.0","github.com/modern-go/concurrent":"0.0.0-20180306012644-bacd9c7ef1dd","github.com/modern-go/reflect2":"1.0.2","github.com/munnerz/goautoneg":"0.0.0-20191010083416-a7dc8b61c822","github.com/namedotcom/go":"0.0.0-20180403034216-08470befbe04","github.com/nrdcg/auroradns":"1.1.0","github.com/nrdcg/desec":"0.6.0","github.com/nrdcg/dnspod-go":"0.4.0","github.com/nrdcg/freemyip":"0.2.0","github.com/nrdcg/goinwx":"0.8.1","github.com/nrdcg/namesilo":"0.2.1","github.com/nrdcg/porkbun":"0.1.1","github.com/oklog/run":"1.0.0","github.com/oracle/oci-go-sdk":"24.3.0+incompatible","github.com/ovh/go-ovh":"1.1.0","github.com/patrickmn/go-cache":"2.1.0+incompatible","github.com/pierrec/lz4":"2.5.2+incompatible","github.com/pkg/errors":"0.9.1","github.com/pmezard/go-difflib":"1.0.0","github.com/pquerna/otp":"1.3.0","github.com/prometheus/client_golang":"1.13.0","github.com/prometheus/client_model":"0.2.0","github.com/prometheus/common":"0.37.0","github.com/prometheus/procfs":"0.8.0","github.com/ryanuber/go-glob":"1.0.0","github.com/sacloud/api-client-go":"0.2.1","github.com/sacloud/go-http":"0.1.2","github.com/sacloud/iaas-api-go":"1.3.2","github.com/sacloud/packages-go":"0.0.5","github.com/scaleway/scaleway-sdk-go":"1.0.0-beta.9","github.com/sirupsen/logrus":"1.8.1","github.com/smartystreets/go-aws-auth":"0.0.0-20180515143844-0c1422d1fdb9","github.com/softlayer/softlayer-go":"1.0.6","github.com/softlayer/xmlrpc":"0.0.0-20200409220501-5f089df7cb7e","github.com/spf13/cast":"1.3.1","github.com/spf13/pflag":"1.0.5","github.com/stretchr/objx":"0.4.0","github.com/stretchr/testify":"1.8.0","github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common":"1.0.490","github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/dnspod":"1.0.490","github.com/transip/gotransip/v6":"6.17.0","github.com/vinyldns/go-vinyldns":"0.9.16","github.com/vultr/govultr/v2":"2.17.2","github.com/yandex-cloud/go-genproto":"0.0.0-20220805142335-27b56ddae16f","github.com/yandex-cloud/go-sdk":"0.0.0-20220805164847-cf028e604997","go.opencensus.io":"0.23.0","go.uber.org/atomic":"1.9.0","go.uber.org/multierr":"1.6.0","go.uber.org/ratelimit":"0.2.0","go.uber.org/zap":"1.21.0","golang.org/x/crypto":"0.1.0","golang.org/x/net":"0.1.0","golang.org/x/oauth2":"0.0.0-20220909003341-f21342109be1","golang.org/x/sys":"0.1.0","golang.org/x/term":"0.1.0","golang.org/x/text":"0.4.0","golang.org/x/time":"0.0.0-20220609170525-579cf78fd858","gomodules.xyz/jsonpatch/v2":"2.2.0","google.golang.org/api":"0.57.0","google.golang.org/genproto":"0.0.0-20220502173005-c8bf987b8c21","google.golang.org/grpc":"1.49.0","google.golang.org/protobuf":"1.28.1","gopkg.in/inf.v0":"0.9.1","gopkg.in/ini.v1":"1.66.6","gopkg.in/ns1/ns1-go.v2":"2.6.5","gopkg.in/square/go-jose.v2":"2.6.0","gopkg.in/yaml.v2":"2.4.0","gopkg.in/yaml.v3":"3.0.1","k8s.io/api":"0.25.2","k8s.io/apiextensions-apiserver":"0.25.0","k8s.io/apimachinery":"0.25.2","k8s.io/client-go":"0.25.2","k8s.io/component-base":"0.25.0","k8s.io/klog/v2":"2.70.1","k8s.io/kube-openapi":"0.0.0-20220803162953-67bda5d908f1","k8s.io/utils":"0.0.0-20220728103510-ee6ede2d64ed","sigs.k8s.io/controller-runtime":"0.13.0","sigs.k8s.io/json":"0.0.0-20220713155537-f223a00ba0e2","sigs.k8s.io/structured-merge-diff/v4":"4.2.3","sigs.k8s.io/yaml":"1.3.0","std":"1.20.14"},"total_functions_found":7}
{"level":"info","ts":1712064479.323352,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:112","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin/server"}
{"level":"info","ts":1712064479.3234062,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:112","msg":"no functions found for probe, removing","name":"database/sql/client"}
{"level":"info","ts":1712064479.323424,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."}
{"level":"info","ts":1712064479.323667,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:190","msg":"Mounting bpffs","allocations_details":{"StartAddr":140509466722304,"EndAddr":140509466984448,"NumCPU":8}}
{"level":"info","ts":1712064479.325669,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:176","msg":"loading probe","name":"google.golang.org/grpc/client"}
{"level":"info","ts":1712064479.964876,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:176","msg":"loading probe","name":"google.golang.org/grpc/server"}
Verifier log: load program: no space left on device:
	(14109 line(s) omitted)
	428: (7b) *(u64 *)(r10 -200) = r6
	429: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	429: (7b) *(u64 *)(r10 -208) = r6
	430: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	430: (7b) *(u64 *)(r10 -216) = r6
	431: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	431: (7b) *(u64 *)(r10 -224) = r6
	432: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	432: (7b) *(u64 *)(r10 -232) = r6
	433: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	433: (7b) *(u64 *)(r10 -240) = r6
	434: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	434: (7b) *(u64 *)(r10 -248) = r6
	435: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	435: (7b) *(u64 *)(r10 -256) = r6
	436: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	436: (7b) *(u64 *)(r10 -264) = r6
	437: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	437: (7b) *(u64 *)(r10 -272) = r6
	438: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	438: (7b) *(u64 *)(r10 -280) = r6
	439: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	439: (7b) *(u64 *)(r10 -288) = r6
	440: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	440: (7b) *(u64 *)(r10 -296) = r6
	441: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	441: (7b) *(u64 *)(r10 -304) = r6
	442: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	442: (7b) *(u64 *)(r10 -312) = r6
	443: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	443: (7b) *(u64 *)(r10 -320) = r6
	444: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	444: (79) r1 = *(u64 *)(r10 -424)
	445: R0=inv(id=0) R1_w=fp-126 R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	445: (05) goto pc-397
	49: R0=inv(id=0) R1_w=fp-126 R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch1 = str[2 * i + 1];
	49: (71) r3 = *(u8 *)(r1 +0)
	50: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	50: (67) r3 <<= 56
	51: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	51: (c7) r3 s>>= 56
	52: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	52: (bf) r2 = r3
	53: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=603,smin_value=-128,smax_value=127) R3_w=inv(id=603,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	53: (57) r2 &= 15
	54: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=603,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch1 = str[2 * i + 1];
{"level":"error","ts":1712064481.3294368,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:179","msg":"error while loading probes, cleaning up","name":"google.golang.org/grpc/server","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 73: (4f) r5 |= r4: 74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm (truncated, 14207 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:179\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:126\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:151\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
	54: (67) r3 <<= 32
	55: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320,var_off=(0x0; 0xffffffff00000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	55: (77) r3 >>= 32
	56: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	56: (bf) r4 = r3
	57: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	57: (77) r4 >>= 6
	58: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	58: (0f) r2 += r4
	59: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	59: (77) r3 >>= 3
	60: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=0,umax_value=536870911,var_off=(0x0; 0x1fffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	60: (57) r3 &= 8
	61: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; out[i] = (nib0 << 4) | nib1;
	61: (4f) r2 |= r3
	62: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch0 = str[2 * i];
	62: (71) r3 = *(u8 *)(r1 -1)
	63: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	63: (67) r3 <<= 56
	64: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	64: (c7) r3 s>>= 56
	65: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	65: (bf) r4 = r3
	66: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=605,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	66: (67) r4 <<= 32
	67: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320,var_off=(0x0; 0xffffffff00000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	67: (77) r4 >>= 32
	68: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	68: (bf) r5 = r4
	69: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	69: (77) r5 >>= 6
	70: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	70: (0f) r5 += r3
	71: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	71: (77) r4 >>= 3
	72: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=536870911,var_off=(0x0; 0x1fffffff)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	72: (57) r4 &= 8
	73: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; out[i] = (nib0 << 4) | nib1;
	73: (4f) r5 |= r4
	74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm
	(truncated)
{"level":"info","ts":1712064481.3792746,"logger":"Instrumentation.Manager.google.golang.org/grpc/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/client"}
{"level":"info","ts":1712064481.3793674,"logger":"Instrumentation.Manager.google.golang.org/grpc/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/server"}
{"level":"info","ts":1712064481.3793843,"logger":"Instrumentation.Manager.net/http/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/server"}
{"level":"info","ts":1712064481.3793988,"logger":"Instrumentation.Manager.net/http/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/client"}
{"level":"info","ts":1712064481.3794107,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:204","msg":"Cleaning bpffs"}
{"level":"error","ts":1712064481.3795562,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 73: (4f) r5 |= r4: 74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm (truncated, 14207 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

Co0ker avatar Apr 02 '24 13:04 Co0ker

I think we need to increase the verifier log buffer during load, so that we can see what the actual error is. The not enough space error is hiding the actual verifier issue. @Co0ker what's your kernel version?

grcevski avatar Apr 02 '24 13:04 grcevski

@grcevski "Detected linux kernel version","version":"5.15.0"

Co0ker avatar Apr 02 '24 14:04 Co0ker

Hi @Co0ker, I tried various ways to reproduce this but I can't see the error on my side. I tried on kernel version 5.15.0-101. Can you please set OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and provide the log after a re-run? Thanks!

grcevski avatar Apr 26 '24 21:04 grcevski