jaeger-operator icon indicating copy to clipboard operation
jaeger-operator copied to clipboard

Published 20 hours ago • verified publisher icon jaegertracing

Sidecar injection failures

Open jpkrohling opened this issue 3 years ago • 18 comments

When installing the simple-prod example in the monitoring namespace, the following is shown in the logs:

INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.810+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.868+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.900+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.918+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.970+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:03.993+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1123] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1123] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:04.018+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1124] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1124] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1147] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1148] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:28.058+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1148] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:28.084+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1148] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:28.105+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1148] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:28.120+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1148] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1150] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
ERRO[1150] failed to update deployment with sidecar      error="Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
2021-12-02T11:29:30.589+0100	ERROR	controller-runtime.manager.controller.deployment	Reconciler error	{"reconciler group": "apps", "reconciler kind": "Deployment", "name": "simple-prod-query", "namespace": "monitoring", "error": "Operation cannot be fulfilled on deployments.apps \"simple-prod-query\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/jpkroehling/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
INFO[1150] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1150] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1157] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring
INFO[1157] injecting Jaeger Agent sidecar                jaeger=simple-prod jaeger-namespace=monitoring name=simple-prod-query namespace=monitoring

jpkrohling avatar Dec 02 '21 10:12 jpkrohling

Which version are you using? 1.28.0? or is this with master?

rubenvp8510 avatar Dec 02 '21 15:12 rubenvp8510

Latest master as of now (e0c53a526df97d83c6ea637fb52586bdb62ca895)

jpkrohling avatar Dec 02 '21 15:12 jpkrohling

I was debugging this issue, and it seems like this happens because we changed the way we auto instrumented the query deployment, when the deployment is starting it mutates its state, and when we try to update the deployment with the sidecar, it throws this kind of errors. I saw this before in other injection scenarios as well.

I can get the object one more time before the update to make sure I have the latest version, but I find this problematic and hard to solve. I think the only definitive solution will be to use webhook mutator.

rubenvp8510 avatar Dec 02 '21 21:12 rubenvp8510

same error in 1.29.0

time="2022-01-11T07:08:37Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=jaeger jaeger-namespace=observability name=jaeger-query namespace=observability

time="2022-01-11T07:08:37Z" level=error msg="failed to update deployment with sidecar" error="Operation cannot be fulfilled on deployments.apps \"jaeger-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=jaeger jaeger-namespace=observability name=jaeger-query namespace=observability

2022-01-11T07:08:37.655Z ERROR controller-runtime.manager.controller.deployment Reconciler error {"reconciler group": "apps", "reconciler kind": "Deployment", "name": "jaeger-query", "namespace": "observability", "error": "Operation cannot be fulfilled on deployments.apps \"jaeger-query\": the object has been modified; please apply your changes to the latest version and try again"}

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214

time="2022-01-11T07:08:38Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=jaeger jaeger-namespace=observability name=jaeger-query namespace=observability

time="2022-01-11T07:08:38Z" level=error msg="failed to update deployment with sidecar" error="Operation cannot be fulfilled on deployments.apps \"jaeger-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=jaeger jaeger-namespace=observability name=jaeger-query namespace=observability

2022-01-11T07:08:38.082Z ERROR controller-runtime.manager.controller.deployment Reconciler error {"reconciler group": "apps", "reconciler kind": "Deployment", "name": "jaeger-query", "namespace": "observability", "error": "Operation cannot be fulfilled on deployments.apps \"jaeger-query\": the object has been modified; please apply your changes to the latest version and try again"}

QLeelulu avatar Jan 11 '22 07:01 QLeelulu

after upgrading jaeger-operator from 1.29 to 1.32 i get the same error

time="2022-03-22T10:24:07Z" level=error msg="failed to update deployment with sidecar" error="Operation cannot be fulfilled on deployments.apps \"general-query\": the object has been modified; please apply your changes to the latest version and try again" jaeger=general jaeger-namespace=jaeger name=general-query namespace=jaeger 2022-03-22T10:24:07.740Z ERROR controller-runtime.manager.controller.deployment Reconciler error {"reconciler group": "apps", "reconciler kind": "Deployment", "name": "general-query", "namespace": "jaeger", "error": "Operation cannot be fulfilled on deployments.apps \"general-query\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214 time="2022-03-22T10:24:07Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=general jaeger-namespace=jaeger name=general-query namespace=jaeger time="2022-03-22T10:24:08Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=general jaeger-namespace=jaeger name=general-query namespace=jaeger

diogoserrano avatar Mar 22 '22 10:03 diogoserrano

I am also having the same error. I am using the very latest version of Jaeger-operator which is 1.34. Error which i am getting looks like

msg="Operation cannot be fulfilled on deployments.apps "myapp": the object has been modified; please apply your changes to the latest version and try again" component=jaeger-cr-sync 1.6532994235610714e+09 ERROR controller.jaeger Reconciler error {"reconciler group": "jaegertracing.io", "reconciler kind": "Jaeger", "name": "jaeger", "namespace": "observability", "error": "Operation cannot be fulfilled on deployments.apps "myapp": the object has been modified; please apply your changes to the latest version and try again"

Any suggestions what should i do?

arushi09-hub avatar May 23 '22 10:05 arushi09-hub

I was debugging this issue, and it seems like this happens because we changed the way we auto instrumented the query deployment, when the deployment is starting it mutates its state, and when we try to update the deployment with the sidecar, it throws this kind of errors. I saw this before in other injection scenarios as well.

I can get the object one more time before the update to make sure I have the latest version, but I find this problematic and hard to solve. I think the only definitive solution will be to use webhook mutator.

How will a webhook mutator be used here? I think I am not aware of it, of how it will be used.

arushi09-hub avatar May 24 '22 10:05 arushi09-hub

Does the error message show an effect? My guess is, when a deployment is created, it passes through the provided webhook to modify the pod spec. if now a Jaeger instance is created, updated or removed in parallel, the reconciler notices that this deployment needs an update. To realize this update the deployment specification is extended with an annotation. The error message appears because the webhook has already modified the deployment spec. Since we return an error in the reconciler, the deployment is evaluated again shortly afterwards.

frzifus avatar May 24 '22 10:05 frzifus

Probably we can ignore this error and simply trigger the Jaeger reconciler again to be on the safe side.

frzifus avatar May 24 '22 10:05 frzifus

Probably we can ignore this error and simply trigger the Jaeger reconciler again to be on the safe side.

What command can I use to trigger the reconciler again?

arushi09-hub avatar May 24 '22 10:05 arushi09-hub

Does the error message show an effect? My guess is, when a deployment is created, it passes through the provided webhook to modify the pod spec. if now a Jaeger instance is created, updated or removed in parallel, the reconciler notices that this deployment needs an update. To realize this update the deployment specification is extended with an annotation. The error message appears because the webhook has already modified the deployment spec. Since we return an error in the reconciler, the deployment is evaluated again shortly afterwards.

I am unable to see this service on my Jaeger UI which is properly up and running and shows the jaegery-query service also. The pods were perfectly running still i could not see it on the UI. Went through the logs of jaeger-operator and saw this error over here. Assumed that it can be because this error.

arushi09-hub avatar May 24 '22 10:05 arushi09-hub

What command can I use to trigger the reconciler again?

Its done automatically. No action need.

I am unable to see this service on my Jaeger UI which is properly up and running and shows the jaegery-query service also. The pods were perfectly running still i could not see it on the UI. Went through the logs of jaeger-operator and saw this error over here. Assumed that it can be because this error.

I assume there is no correlation. But i will look into it.

frzifus avatar May 24 '22 11:05 frzifus

Having same problem - jaeger-operator v1.36.0. Yesterday was updated along with the cluster components. Previously on v1.35.0 there were no problems. Reverting the version didn't fix the problem

violinorg avatar Aug 25 '22 13:08 violinorg

@violinorg, could you provide some log output?

iblancasa avatar Aug 25 '22 19:08 iblancasa

Probably the log output is similar to the other reports. Looks like the logging is causing confusion, maybe we should change the reported message and/or only log on debug level.

What do you think?

frzifus avatar Aug 25 '22 22:08 frzifus

If it is not raising a real issue...

iblancasa avatar Aug 25 '22 22:08 iblancasa

@iblancasa yes of course

time="2022-08-26T12:40:04Z" level=error msg="Operation cannot be fulfilled on deployments.apps \"r882-dcc95-observability-tracing-demo\": the object has been modified; please apply your changes to the latest version an │
│ 1.6615176046172814e+09    ERROR    Reconciler error    {"controller": "jaeger", "controllerGroup": "jaegertracing.io", "controllerKind": "Jaeger", "jaeger": {"name":"observability-t1260-3f32c-jaeger-operator-jaeger","n │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                                                                                                                                   │
│     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273

violinorg avatar Aug 26 '22 12:08 violinorg

@frzifus I'm not sure but it seems this message is not printed by Jaeger Operator but by the inherit logger.

iblancasa avatar Aug 26 '22 13:08 iblancasa