dotnet-sdk icon indicating copy to clipboard operation
dotnet-sdk copied to clipboard

Sidecar crashes when starting workflow using StartWorkflowAsync

Open hsinghdeol-wpay opened this issue 2 years ago • 4 comments

Expected Behavior

Workflow should be started gracefully

Actual Behavior

sidecar crashes with following stacktrace when using dapr 1.12.0 version

[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: panic: runtime error: index out of range [4] with length 3
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]:
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: goroutine 238 [running]:
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/diagnostics.(*httpMetrics).convertPathToMetricLabel(0x1?, {0xc001624618, 0x13})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/diagnostics/http_monitoring.go:288 +0x4ac
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/diagnostics.(*httpMetrics).ClientRequestStarted(0xc0001aafc0, {0x6a2c4d0, 0xc00166e960}, {0xc001482217, 0x4}, {0xc001624618?, 0x4362d17?}, 0x0)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/diagnostics/http_monitoring.go:153 +0xa7
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/channel/http.(*Channel).invokeMethodV1(0xc000be4880, {0x6a2c4d0, 0xc00166e960}, 0xc000704a80, {0x0, 0x0})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/channel/http/http_channel.go:231 +0x1e5
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/channel/http.(*Channel).InvokeMethod(0x6a2c578?, {0x6a2c4d0?, 0xc00166e960?}, 0x1d?, {0x0?, 0xc000d1f3c0?})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/channel/http/http_channel.go:166 +0x299
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/grpc.(*api).CallLocalStream(0xc00120eb40, {0x6a45e58?, 0xc001620ae0})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/grpc/api_daprinternal.go:198 +0x77a
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/proto/internals/v1._ServiceInvocation_CallLocalStream_Handler({0x5ec6640?, 0xc00120eb40}, {0x6a3af00?, 0xc0016500e0})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/proto/internals/v1/service_invocation_grpc.pb.go:180 +0x9f
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/diagnostics.(*grpcMetrics).StreamingClientInterceptor.func1({0x5ec6640, 0xc00120eb40}, {0x6a3af00?, 0xc0016500e0?}, 0xc00164a7b0, 0x61dc7d8)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/diagnostics/grpc_monitoring.go:315 +0x1c5
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func2.1({0x5ec6640?, 0xc00120eb40?}, {0x6a3af00?, 0xc0016500e0?})
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:83 +0x4f
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/dapr/dapr/pkg/diagnostics.GRPCTraceStreamServerInterceptor.func1({0x5ec6640, 0xc00120eb40}, {0x6a3b8d8, 0xc0001f5770?}, 0xc00164a7b0, 0xc001670000)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: D:/a/dapr/dapr/pkg/diagnostics/grpc_tracing.go:155 +0x4cc
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func2({0x5ec6640, 0xc00120eb40}, {0x6a3b8d8, 0xc0001f5770}, 0xc00164a7b0, 0xc001620ac0?)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:86 +0x14e
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc00149c1e0, {0x6a4b4e0, 0xc001654680}, 0xc001634480, 0xc001499e60, 0x9fdee00, 0x0)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/google.golang.org/[email protected]/server.go:1661 +0x1383
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: google.golang.org/grpc.(*Server).handleStream(0xc00149c1e0, {0x6a4b4e0, 0xc001654680}, 0xc001634480, 0x0)
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/google.golang.org/[email protected]/server.go:1741 +0x9f0
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: google.golang.org/grpc.(*Server).serveStreams.func1.1()
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/google.golang.org/[email protected]/server.go:982 +0x98
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: created by google.golang.org/grpc.(*Server).serveStreams.func1
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: C:/Users/runneradmin/go/pkg/mod/google.golang.org/[email protected]/server.go:980 +0x18c
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: The daprd process exited with error code: exit status 2
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]:
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: terminated signal received: shutting down
[wpay-connect-transaction-management-workflow-dapr_adc38c35-3]: Error exiting Dapr: exit status 2

Steps to Reproduce the Problem

Try to execute following line return await _daprClient.StartWorkflowAsync( workflowComponent: "dapr", workflowName: "blah" instanceId: "12345565656", input: "sdfd"); "blah" is a valid workflow with activities

Release Note

RELEASE NOTE:

hsinghdeol-wpay avatar Oct 22 '23 19:10 hsinghdeol-wpay

@hsinghdeol-wpay Thanks for the report! Which version of .NET SDK are you using?

While I've not been able to reproduce this issue exactly--I can get it to panic in the same place with a malformed workflow API URL, but not actually bring the entire process down--I do think I've found the issue, in the way HTTP diagnostic data is generated in the Dapr Runtime.

philliphoff avatar Oct 23 '23 18:10 philliphoff

actually bring the entire process down--I do think I've found the issue, in the way HTTP diagnostic data is generated in the Dapr Runtime.

It is 1.12. I have actually found the issue with my application. Coincidently I have a controller that triggers dapr workflows and it is served at 'api/workflows/start'. go middlewares assumes it is path related to workflows within dapr and is malformed

hsinghdeol-wpay avatar Oct 23 '23 23:10 hsinghdeol-wpay

@hsinghdeol-wpay - Based on your last comment, is this resolved, or do you still need assistance?

halspang avatar Nov 29 '23 22:11 halspang

Yes it is resolved

hsinghdeol-wpay avatar Nov 30 '23 20:11 hsinghdeol-wpay