[Bug] Several services crashing when deploying v.0.24.0

[vishwa-trulioo]

What are you really trying to do?

I attempted updating the Temporal deployment from Helm chart version 0.22.1 to 0.24.0. Several of the services starts crashing.

Details: Current Helm chart version: v0.22.1 Expected version: v0.24.0 Storage backend: MySQL/Aurora

Error summary:

sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9.

Kubernetes env details:

% kubectl --namespace=temporal get pods -l "app.kubernetes.io/instance=temporal"
NAME                                  READY   STATUS             RESTARTS        AGE
temporal-admintools-c5b5554f8-6hdm2   1/1     Running            0               11m
temporal-frontend-5dd4b6cd49-qrtkj    1/1     Running            0               17m
temporal-frontend-7895bcd6c6-fb7mn    0/1     CrashLoopBackOff   7 (4s ago)      11m
temporal-frontend-7895bcd6c6-npqsj    0/1     CrashLoopBackOff   7 (20s ago)     11m
temporal-history-774b8bf6b9-hfw9c     0/1     CrashLoopBackOff   2 (13s ago)     35s
temporal-history-774b8bf6b9-r6zqw     0/1     CrashLoopBackOff   7 (35s ago)     11m
temporal-history-fb6bb49cd-kkdvb      1/1     Running            0               17m
temporal-matching-65b59c5868-wtw9z    1/1     Running            0               17m
temporal-matching-758f4c7865-76ztd    0/1     CrashLoopBackOff   7 (25s ago)     11m
temporal-matching-758f4c7865-hmrt5    0/1     CrashLoopBackOff   5 (2m19s ago)   5m29s
temporal-web-956f445dd-tx6mg          1/1     Running            0               11m
temporal-worker-5d8dc58949-9clhc      1/1     Running            0               17m
temporal-worker-5d8dc58949-dhnvn      1/1     Running            0               17m
temporal-worker-d849c84bb-jqxbg       0/1     CrashLoopBackOff   7 (14s ago)     11m

Error details:

• All crashing pods show the an error similar to this,

kubectl -n temporal logs temporal-frontend-7895bcd6c6-58mfg
TEMPORAL_ADDRESS is not set, setting it to 10.96.36.97:7233
2023/07/06 18:17:47 Loading config; env=docker,zone=,configDir=config
2023/07/06 18:17:47 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2023-07-06T18:17:47.005Z","msg":"Build info.","git-time":"2023-07-01T00:28:10.000Z","git-revision":"2f4b63623bd895eeea98df72115cdb26b4939e39","git-modified":true,"go-arch":"amd64","go-os":"linux","go-version":"go1.20.5","cgo-enabled":false,"server-version":"1.21.1","debug-mode":false,"logging-call-at":"main.go:148"}
{"level":"info","ts":"2023-07-06T18:17:47.006Z","msg":"Updated dynamic config","logging-call-at":"file_based_client.go:195"}
{"level":"warn","ts":"2023-07-06T18:17:47.006Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"main.go:178"}
[Fx] PROVIDE	*pprof.PProfInitializerImpl <= go.temporal.io/server/common/pprof.NewInitializer()
[Fx] PROVIDE	*temporal.ServerImpl <= go.temporal.io/server/temporal.NewServerFxImpl()
[Fx] PROVIDE	*temporal.serverOptions <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	chan interface {} <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	temporal.synchronizationModeParams <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	*config.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	*config.PProf <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	log.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	resource.ServiceNames <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	resource.NamespaceLogger <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	resolver.ServiceResolver <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	client.AbstractDataStoreFactory <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	searchattribute.Mapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	[]grpc.UnaryServerInterceptor <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	authorization.Authorizer <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	authorization.ClaimMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	authorization.JWTAudienceMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	log.Logger <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	client.FactoryProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	dynamicconfig.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	encryption.TLSConfigProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	*client.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	client.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	metrics.Handler <= go.temporal.io/server/temporal.ServerOptionsProvider()
[Fx] PROVIDE	[]trace.SpanExporter <= go.temporal.io/server/temporal.glob..func2()
[Fx] PROVIDE	client.FactoryProviderFn <= go.temporal.io/server/temporal.PersistenceFactoryProvider()
[Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.HistoryServiceProvider()
[Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.MatchingServiceProvider()
[Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.FrontendServiceProvider()
[Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.InternalFrontendServiceProvider()
[Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.WorkerServiceProvider()
[Fx] PROVIDE	*cluster.Config <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
[Fx] PROVIDE	config.Persistence <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
[Fx] SUPPLY	[]temporal.ServerOption
[Fx] PROVIDE	fx.Lifecycle <= go.uber.org/fx.New.func1()
[Fx] PROVIDE	fx.Shutdowner <= go.uber.org/fx.(*App).shutdowner-fm()
[Fx] PROVIDE	fx.DotGraph <= go.uber.org/fx.(*App).dotGraph-fm()
[Fx] ERROR		Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2
	/go/pkg/mod/go.uber.org/[email protected]/module.go:218:
failed to build fxevent.Logger:
could not build arguments for function "go.temporal.io/server/temporal".glob..func8
	/home/builder/temporal/temporal/fx.go:1004:
failed to build log.Logger:
received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
	/home/builder/temporal/temporal/fx.go:166:
sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/[email protected]/module.go:218): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1004): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:166): sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9.

Notes:

• I did verify that schema is updated to fit the required temporal version (v1.21.1)
• I even did a clean new install. It still produces the same errors.

Minimal Reproduction

Environment/Versions

• OS and processor: Linux
• Temporal Version: 1.21.1
• Hosted environment: Kubernetes/EKS

Additional context

• I did verify that schema is updated to fit the required temporal version (v1.21.1)
• I even did a clean new install. It still produces the same errors.