Temporal services are not starting when deployed in istio enabled namespace

[mmlk09]

What are you really trying to do?

Trying to deploy temporal in istio enabled namespace along with PostgreSQL DB on a Kubernetes cluster. Deployment is done using helm charts available at - https://github.com/temporalio/helm-charts

Describe the bug

Post deployment history, matching and worker PODs are continuously restarting with CrashLoopBackOff with below mentioned logs.

Here is status of PODs: NAME READY STATUS RESTARTS AGE temporaldb-7d7f9c67b8-qqxbb 2/2 Running 0 22m temporaltest-admintools-7bf55d4b8f-t47m2 2/2 Running 0 16m temporaltest-frontend-546c569b9b-g4gv4 2/2 Running 0 16m temporaltest-history-8648bd7db9-xsnz4 1/2 CrashLoopBackOff 6 (74s ago) 16m temporaltest-matching-bc9f69487-9vc95 1/2 CrashLoopBackOff 6 (52s ago) 16m temporaltest-web-67d46c8748-zf4xn 2/2 Running 0 16m temporaltest-worker-5c58f8cc58-c6jwd 1/2 Error 6 (4m32s ago) 16m

LOGs: {"level":"info","ts":"2022-12-07T17:48:04.721Z","msg":"bootstrap hosts fetched","service":"worker","bootstrap-hostports":"10.4.0.125:6939,10.4.0.124:6933,10.4.0.126:6935,10.4.0.127:6934","logging-call-at":"rpMonitor.go:285"} {"level":"error","ts":"2022-12-07T17:48:46.557Z","msg":"unable to bootstrap ringpop. retrying","service":"worker","error":"join duration of 41.835509067s exceeded max 30s","logging-call-at":"ringpop.go:109","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap.func1\n\t/home/builder/temporal/common/membership/ringpop.go:109\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:170\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:171\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap\n\t/home/builder/temporal/common/membership/ringpop.go:114\ngo.temporal.io/server/common/membership.(*RingPop).Start\n\t/home/builder/temporal/common/membership/ringpop.go:84\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:135\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:268\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/[email protected]/internal/lifecycle/lifecycle.go:120\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/[email protected]/internal/lifecycle/lifecycle.go:85\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/[email protected]/app.go:683\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/[email protected]/app.go:773"} {"level":"fatal","ts":"2022-12-07T17:48:46.557Z","msg":"unable to bootstrap ringpop. exhausted all retries","service":"worker","error":"join duration of 41.835509067s exceeded max 30s","logging-call-at":"ringpop.go:116","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap\n\t/home/builder/temporal/common/membership/ringpop.go:116\ngo.temporal.io/server/common/membership.(*RingPop).Start\n\t/home/builder/temporal/common/membership/ringpop.go:84\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:135\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:268\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/[email protected]/internal/lifecycle/lifecycle.go:120\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/[email protected]/internal/lifecycle/lifecycle.go:85\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/[email protected]/app.go:683\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/[email protected]/app.go:773"}

Environment/Versions

• OS and processor: GCP VM - Intel(R) Xeon(R) CPU @ 2.80GHz with 4 procs and 32 GB RAM, Ubuntu 20.04.5 LTS
• Temporal Version: 1.18.5
• Are you using Docker or Kubernetes or building Temporal from source? Kubernetes (helm charts)

Additional context

Temporal deployment is done on KiND based Kubernetes cluster KiND version: v0.17.0 K8s version: 1.25.3 Istio version: 1.16.0