[Bug] DB errors after running server for a while

[feedmeapples]

What are you really trying to do?

Describe the bug

{"level":"error","ts":"2023-02-22T08:56:41.887-0800","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: SQL logic error: no such table: namespaces (1)","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\tgo.temporal.io/[email protected]/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\tgo.temporal.io/[email protected]/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\tgo.temporal.io/[email protected]/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\tgo.temporal.io/[email protected]/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\tgo.temporal.io/[email protected]/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\tgo.temporal.io/[email protected]/common/namespace/registry.go:357\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\tgo.temporal.io/[email protected]/internal/goro/goro.go:64"}

{"level":"error","ts":"2023-02-22T08:56:41.892-0800","msg":"Operation failed with internal error.","error":"GetTaskQueue operation failed. Failed to check if task queue default-worker-tq of type Workflow existed. Error: SQL logic error: no such table: task_queues (1)","operation":"GetTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\tgo.temporal.io/[email protected]/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTaskQueue.func1\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:567\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTaskQueue\n\tgo.temporal.io/[email protected]/common/persistence/persistenceMetricClients.go:569\ngo.temporal.io/server/service/matching.(*taskQueueDB).takeOverTaskQueueLocked\n\tgo.temporal.io/[email protected]/service/matching/db.go:123\ngo.temporal.io/server/service/matching.(*taskQueueDB).RenewLease\n\tgo.temporal.io/[email protected]/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry.func1\n\tgo.temporal.io/[email protected]/service/matching/taskWriter.go:302\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\tgo.temporal.io/[email protected]/common/backoff/retry.go:199\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry\n\tgo.temporal.io/[email protected]/service/matching/taskWriter.go:306\ngo.temporal.io/server/service/matching.(*taskWriter).initReadWriteState\n\tgo.temporal.io/[email protected]/service/matching/taskWriter.go:131\ngo.temporal.io/server/service/matching.(*taskWriter).taskWriterLoop\n\tgo.temporal.io/[email protected]/service/matching/taskWriter.go:221\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\tgo.temporal.io/[email protected]/internal/goro/goro.go:64"}

Minimal Reproduction

run temporal server start-dev -f ~/db for a ~day

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]
• Temporal Version: [e.g. 1.14.0?] and/or SDK version
• Are you using Docker or Kubernetes or building Temporal from source?

Additional context

[josh-berry]

Will be fixed by bumping Server to 1.26 once it's available.