Operator resource update failed after authentication is disabled

Open cdmikechen opened this issue 3 years ago • 1 comments

I created a SolrCloud and enable authentication, and then I closed authentication for some reason. When I changed CR, I found that operator can not work.

2022-06-13T02:04:45.551Z	INFO	controller-runtime.manager.controller.solrcloud	Starting workers	{"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "worker count": 1}
2022-06-13T02:04:45.871Z	ERROR	controller-runtime.manager.controller.solrcloud.ManagedUpdateSelector	Error retrieving cluster status, delaying pod update selection	{"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "name": "solr-project1", "namespace": "project-1", "error": "Recieved bad response code of 401 from solr with response: <html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n<title>Error 401 Authentication failed, Response code: 401</title>\n</head>\n<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>\n<table>\n<tr><th>URI:</th><td>/solr/admin/collections</td></tr>\n<tr><th>STATUS:</th><td>401</td></tr>\n<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>\n<tr><th>SERVLET:</th><td>default</td></tr>\n</table>\n\n</body>\n</html>\n"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/apache/solr-operator/controllers/util.DeterminePodsSafeToUpdate
	/workspace/controllers/util/solr_update_util.go:122
github.com/apache/solr-operator/controllers.(*SolrCloudReconciler).Reconcile
	/workspace/controllers/solrcloud_controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99
2022-06-13T02:04:47.203Z	ERROR	controller-runtime.manager.controller.solrcloud.ManagedUpdateSelector	Error retrieving cluster status, delaying pod update selection	{"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "name": "solr-project1", "namespace": "project-1", "error": "Recieved bad response code of 401 from solr with response: <html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n<title>Error 401 Authentication failed, Response code: 401</title>\n</head>\n<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>\n<table>\n<tr><th>URI:</th><td>/solr/admin/collections</td></tr>\n<tr><th>STATUS:</th><td>401</td></tr>\n<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>\n<tr><th>SERVLET:</th><td>default</td></tr>\n</table>\n\n</body>\n</html>\n"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/apache/solr-operator/controllers/util.DeterminePodsSafeToUpdate
	/workspace/controllers/util/solr_update_util.go:122
github.com/apache/solr-operator/controllers.(*SolrCloudReconciler).Reconcile
	/workspace/controllers/solrcloud_controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99

I used solr-operator v0.5.1, and I think it is a bug?

Jun 13 '22 02:06 cdmikechen

Well the operator doesn't have a way to remove Auth from a SolrCloud, at least not yet.

One fix you can do yourself manually is do a kubectl exec --stdin --tty <solr-cloud-pod-name> bin/solr auth disable -z <zookeeper-connection-string>.

This is something we could fix in the Solr Operator though

Jun 22 '22 22:06 HoustonPutman