Support for Dapr with PubSub component sidecar debugging - dapr in error state

[katriendg]

Describe the bug

I'm looking for some guidance on achieving K8S bridge debugging with Dapr sidecar leveraging a PubSub component. Using .NET and the setup is similar to a simplified version of this one: https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub My application is subscribing to a topic 'A' and publishing to a topic 'B'. Just one app interacting with the pubsub component.

Because of the setup of Pods with Dapr, bridge to Kubernetes configuration needs to be able to select the right container in the Pod (as it has multiple containers). This was not working at first but it's now correctly working by having a Service and a port in the Pod spec that maps the right container to pick for debugging. In my case the below references to port 5050 make it work. The application is using gRPC to interact with Dapr.

Configuration of the Pod container:

... sections omitted

 template:
    metadata:
      labels:
        app: myappname
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "myappname"
        dapr.io/enable-api-logging: "true"
        dapr.io/app-port: "5050"
        dapr.io/app-protocol: "grpc"
        dapr.io/log-level: "debug"
spec:
      containers:
      - name: myappname
        image: myregistry/myappname:0.1
        imagePullPolicy: Always
        env:
        - name: Logging__LogLevel__Default
          value: Debug
        ports:
        - containerPort: 5050
...

Configuration of the service:

kind: Service
apiVersion: v1
metadata:
  name: myappname-workload-svc
  labels:
    app: myappname
spec:
  selector:
    app: myappname
  ports:
  - protocol: TCP
    port: 5050
    targetPort: 5050 
    name: grpcmain
  type: ClusterIP

KubernetesLocalProcessConfig.yaml config:

version: 0.1
env:
  - name: DEBUG_MODE 
    value: "true"
  - name: DAPR_GRPC_PORT
    value: "50001"

Task config (pointing to port 5050 for the right container selection within pod):

{
	"label": "bridge-to-kubernetes.resource",
	"type": "bridge-to-kubernetes.resource",
	"resource": "myappname-workload-svc",
	"resourceType": "service",
	"ports": [
		5050
	],
	"targetCluster": "k3d-devcluster",
	"targetNamespace": "default",
	"useKubernetesServiceEnvironmentVariables": true
}

The debugging session starts well, and my custom app enters debug mode. I should then be subscribing to a topic using the PubSub component, however this code is never reached as the Dapr sidecar enters an error state. Sharing a normal log and the error log:

Normal running Daprd container logs (without debug session with K8S bridge enabled yet)

time="2023-10-27T09:14:28.263676363Z" level=debug msg="established connection to placement service at dns:///dapr-placement-server.dapr-system.svc.cluster.local:50005" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:14:28.264382502Z" level=debug msg="placement order received: lock" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:14:28.264465159Z" level=debug msg="placement order received: update" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:14:28.264492278Z" level=info msg="placement tables updated, version: 0" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:14:28.2645005Z" level=debug msg="placement order received: unlock" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:14:28.388730781Z" level=info msg="app is subscribed to the following topics: [A] through pubsub=mypubsub" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:14:28.388797988Z" level=debug msg="subscribing to topic='A' on pubsub='mypubsub'" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:14:28.389126203Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 206ms" app_id=myappname instance=myappname-568797cc9b-kw9ks scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:14:36.442713599Z" level=info msg="HTTP API Called" app_id=myappname instance=myappname-568797cc9b-kw9ks method="GET /v1.0/healthz" scope=dapr.runtime.http-info type=log useragent=kube-probe/1.25 ver=1.10.9

Daprd container logs - shows container error once K8S bridge debug session is initialized

time="2023-10-27T09:08:29.039540887Z" level=info msg="placement tables updated, version: 0" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:08:29.03961823Z" level=debug msg="placement order received: unlock" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime.actor.internal.placement type=log ver=1.10.9
time="2023-10-27T09:08:29.206885422Z" level=info msg="Error processing operation DaprBuiltInInitializationRetries. Retrying in 456.489313ms…" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:08:29.206924869Z" level=debug msg="Error for operation DaprBuiltInInitializationRetries was: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: EOF\"" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:08:31.762185717Z" level=error msg="error getting topic list from app: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: EOF\"" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:08:31.762238075Z" level=error msg="error occurred while beginning pubsub mypubsub: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: EOF\"" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime type=log ver=1.10.9
time="2023-10-27T09:08:31.76225593Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 5227ms" app_id=myappname instance=myappname-6ff6dfd7c9-klljr scope=dapr.runtime type=log ver=1.10.9

The Dapr container in the pod is creating a connection to the bridge agent container in the pod, and not to my local running version of the code as it seems port forwarding within a pod (not using an external svc) is not being forwarded. I already tried out setting up a 3rd sidecar container in the same Pod with an nginx to be able to do some telnet and curl commands and within the pod my :5050 endpoint is available and exposed through the bridge container.

Additionally I am wondering how my local debug session will be able to access the Dapr sidecar to publish a new message with daprClient.PublishEventAsync as the sidecar is probably also not forwarded in the opposite direction (from my container to the sidecar).

Sharing the logs of the agent (in my pod, it's using image bridgetokubernetes.azurecr.io/lpkremoteagent:1.3.2). Note the agent is also logging an error in the last line. The actual log was bigger but repeating the last lines.

2023-10-26T11:02:15.7299133Z | RemoteAgent | TRACE | ReversePortForwardConnector created for port 5050\nOperation context: <json>{"clientRequestId":"0702b0fc-581f-4328-9fff-747876b9ca57","correlationId":"b90d2632-af0c-4c8e-8973-4f265e82b7e71698239043742:f0ac2b986a2f:3b1c4eb3f741","requestId":null,"userSubscriptionId":null,"startTime":"2023-10-26T11:02:12.8756594+00:00","userAgent":"RemoteAgent/1.0.20230929.1","requestHttpMethod":null,"requestUri":null,"version":"1.0.20230929.1","requestHeaders":{},"loggingProperties":{"ApplicationName":"RemoteAgent","DeviceOperatingSystem":"Linux 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023","Framework":".NET 7.0.11","ProcessId":1,"TargetEnvironment":"Production"}}</json>
2023-10-26T11:02:15.7445569Z | RemoteAgent | TRACE | ReversePortForwardConnector start listening on port 5050
2023-10-26T11:02:15.7886293Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 5050 accepted incoming request as stream 1.
2023-10-26T11:02:15.7894726Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 5050 accepted incoming request as stream 2.
2023-10-26T11:02:15.7915012Z | RemoteAgent | TRACE | AgentHub connnected for 5050, id 1
2023-10-26T11:02:15.7922040Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 1 returns 0 bytes.
2023-10-26T11:02:15.7923766Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 1.
2023-10-26T11:02:15.7928424Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 1
2023-10-26T11:02:15.7931140Z | RemoteAgent | TRACE | AgentHub connnected for 5050, id 2
2023-10-26T11:02:15.7969806Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 2 returns 33 bytes.
2023-10-26T11:02:15.7985616Z | RemoteAgent | TRACE | AgentHub received for 5050, id 2, size 33
2023-10-26T11:02:15.8065116Z | RemoteAgent | TRACE | AgentHub closed for 5050, id 1
2023-10-26T11:02:15.8973681Z | RemoteAgent | TRACE | AgentHub disconnect for 5050, id 2\nOperation context: <json>{"clientRequestId":"0702b0fc-581f-4328-9fff-747876b9ca57","correlationId":"b90d2632-af0c-4c8e-8973-4f265e82b7e71698239043742:f0ac2b986a2f:3b1c4eb3f741","requestId":null,"userSubscriptionId":null,"startTime":"2023-10-26T11:02:12.8756594+00:00","userAgent":"RemoteAgent/1.0.20230929.1","requestHttpMethod":null,"requestUri":null,"version":"1.0.20230929.1","requestHeaders":{},"loggingProperties":{"ApplicationName":"RemoteAgent","DeviceOperatingSystem":"Linux 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023","Framework":".NET 7.0.11","ProcessId":1,"TargetEnvironment":"Production"}}</json>
2023-10-26T11:02:15.8977029Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 2
2023-10-26T11:02:15.8978804Z | RemoteAgent | TRACE | AgentHub closed for 5050, id 2
2023-10-26T11:02:15.9033039Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync exception 'Unable to read data from the transport connection: Operation canceled.' when invoking handler. Close.

Mention the platform you are using

• Devcontainer with Visual Studio Code
• K3D cluster running in the devcontainer

To Reproduce Steps to reproduce the behavior:

1. Setup a solution with dapr and pubsub
2. Configure Bridge
3. Run debug mode and validate Dapr container logs
4. See error

Expected behavior Port forwarding seems to be missing between sidecards in a Pod with multiple containers. When calling other kubernetes services with a Service configuration this works when these are defined through the KubernetesLocalProcessConfig.yaml file.

Logs: Please see some of the logs shared above.

Desktop (please complete the following information):

• OS: Windows 11 WSL 2 - Ubuntu 20.04