nack icon indicating copy to clipboard operation
nack copied to clipboard
verified publisher icon nats-io

Recreating a workqueue stream with the same name sometimes ends in a crash loop

Open Koalk opened this issue 6 months ago • 0 comments

What version were you using?

nats:2.11.4-alpine nack-0.28.1

What environment was the server running in?

Kubernetes cluster in google cloud using ArgoCD for deployments

Is this defect reproducible?

It is not consistent but it happens often enough that is not too hard to reproduce. Create a stream with retention set to workqueue, remove it and create it again. Repeat until you hit the issue. For example:

apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: foo
spec:
  name: foo
  subjects: ["foo", "foo.>"]  
  storage: file
  retention: workqueue

Given the capability you are leveraging, describe your expectation?

No matter the amount of times I remove and recreate a stream, it should keep NATS consistent with the kubernetes resources created.

Given the expectation, what is the defect you are observing?

When the resource is removed and readed multiple times, it eventually gets into this inconsistent state. It seems it deletes successfully the stream but the next time it needs to create it, its trying to update it instead. Furthermore, the update seems to be trying to change the retention policy which should be the same even if it didn't actually deleted the previous version of the stream. Deleting the stream manually again and recreating it will take it out of that state.

Logs from the Nack controller for a stream called preview-api-pr-9739-metrics-t-parts-request:

I0619 15:46:55.001625       1 controller.go:312] stream preview/preview-api-pr-9739-metrics-t-parts-request might be missing, looking it up...
I0619 15:46:55.008706       1 controller.go:318] stream preview/preview-api-pr-9739-metrics-t-parts-request was not found anymore, deleting from JetStream
I0619 15:46:55.009577       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"002ba751-12b7-4132-9a16-9de36962740d", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687798695", FieldPath:""}): type: 'Normal' reason: 'Deleting' Deleting stream "preview-api-pr-9739-metrics-t-parts-request"
I0619 15:46:55.444148       1 controller.go:325] deleted stream preview/preview-api-pr-9739-metrics-t-parts-request from JetStream
I0619 15:47:49.433774       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687832416", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:49.460866       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.397968       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687833945", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.422765       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.470586       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834412", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.492432       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.531079       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834423", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.552127       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.601313       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834432", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.617264       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.703387       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834445", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.719997       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:53.892697       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834461", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:53.907227       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:54.236618       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834483", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:54.268826       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:54.912878       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834514", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:54.928130       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:47:58.033916       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834592", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:47:58.054592       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0619 15:48:01.515188       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687834832", FieldPath:""}): type: 'Normal' reason: 'Updating' Updating stream "preview-api-pr-9739-metrics-t-parts-request"
E0619 15:48:01.532121       1 controller.go:682] "Unhandled Error" err="failed to process stream: failed to update stream \"preview-api-pr-9739-metrics-t-parts-request\": stream configuration update can not change retention policy to/from workqueue (10052)" logger="UnhandledError"
I0620 08:15:09.301391       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1689062759", FieldPath:""}): type: 'Normal' reason: 'Deleting' Deleting stream "preview-api-pr-9739-metrics-t-parts-request"
I0620 08:15:24.490709       1 controller.go:312] stream preview/preview-api-pr-9739-metrics-t-parts-request might be missing, looking it up...
I0620 08:15:24.499265       1 controller.go:318] stream preview/preview-api-pr-9739-metrics-t-parts-request was not found anymore, deleting from JetStream
I0620 08:15:24.500987       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"23ba115c-23e8-474f-9108-794e32a402a2", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1687835031", FieldPath:""}): type: 'Normal' reason: 'Deleting' Deleting stream "preview-api-pr-9739-metrics-t-parts-request"
I0620 08:15:24.502385       1 controller.go:325] deleted stream preview/preview-api-pr-9739-metrics-t-parts-request from JetStream
I0620 08:15:24.656230       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"4420c456-d185-4aff-a014-7ca642f16099", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1689063214", FieldPath:""}): type: 'Normal' reason: 'Creating' Creating stream "preview-api-pr-9739-metrics-t-parts-request"
I0620 08:15:24.735398       1 event.go:377] Event(v1.ObjectReference{Kind:"Stream", Namespace:"preview", Name:"preview-api-pr-9739-metrics-t-parts-request", UID:"4420c456-d185-4aff-a014-7ca642f16099", APIVersion:"jetstream.nats.io/v1beta2", ResourceVersion:"1689063214", FieldPath:""}): type: 'Normal' reason: 'Created' Created stream "preview-api-pr-9739-metrics-t-parts-request"

Koalk avatar Jun 20 '25 10:06 Koalk