Use zkStateReader to query for collections and configsets

[swarupdonepudi]

Is your feature request related to a problem? Please describe. I am creating this issue to track the implementation of using zkstatereader to query for collections and other resources directly to zookeeper instead of solr to keep the number of queries to solr low.

Describe the solution you'd like Instead of rewriting the zkstatereader.java monster in golang, how about using the java library itself in the solr operator?

I was able to prototype this out by creating a simple rest interface on top of zkstatereader. You can check it out in solr-zkstatereader-rest repo and on swarupdonepudi/zkreader dockerhub repo

Basically the idea is to run this zkstatereader container as a side car container to the solr-operator and send queries to this container on localhost for all the queries that zkstatereader can answer. This was we dont looseout on the actual code base updates to this library either. We simply bump up the maven dependency version and rebuild the image.

Describe alternatives you've considered The other alternative is to actually implement this in golang. But it is going to take us forever to do that and does not really give any additional value.

Additional context The next step is to create a ci/cd process for the solr-zkstatereader-rest on travis and updating the solr-operator to set this one up as side car container in the solr-operator pod. I will be looking forward for feedback from @HoustonPutman and @sepulworld on this one.

[sepulworld]

This seems like a solid solution! Especially since rewriting zkstatereader.java and a bunch of the imports it has would be daunting, plus after that we would have to keep up with changes that would have to be back ported to this rewrite.

If @HoustonPutman thinks it is worth the time, perhaps submit a PR with the switch over to this zkstatereader sidecar?

[sepulworld]

This sidecar solution could be swapped out later with a Go rewrite solution later if we choose to go down that route. So this is a good first step I think!

[HoustonPutman]

Does this provide additional functionality beyond calling the Collections API of solr?

I think what I was looking for when I made the Go ZkStateReader idea, was that we would be able to use watches and get notifications on state updates. Creating a REST endpoint on top of it wouldn't give us that functionality, as far as I know.

If there is info that we can't get from the Collections API, then I think this would make sense. If not, then I think it would be best to just use that since it's already a REST endpoint for the information.

[swarupdonepudi]

hmmm so you are the one who implemented CloudCollectionsListener @HoustonPutman ... http://opensearch.krugle.org/document/view_filecontent/solr_os/scmi_30.20.55.1_8767/lucene-solr-code/solr/CHANGES.txt#85 Nice work :)

[HoustonPutman]

Haha trip down memory lane, and thanks! But generally that's basically what operators do, they listen on changes in kube object states and send out notifications when something happens. The ZkStateReader does the same thing for internal Solr state, listening on new/deleted collections, updated collection state, live node changes, etc.

I think we could definitely write the functionality into the Solr operator to listen on these things, but it would take a relatively significant amount of work.

We don't need to get 100% of the ZkStateReader functionality from the get-go. We could start by implementing something more basic like the CloudCollectionsListener, mentioned above, or the LiveNodesListener (Though this isn't as useful because we already know which nodes are alive from kube health checks.). Then we can slowly migrate over one feature at a time, replacing Collections API calls with this new go ZKStateReader.

I'm not opposed to the ZkStateReader sidecar method as a first pass, if it gives us something more than what the Collections API already gives us. If not, then I think any work in that area would be better spent adding watch functionality in go, so that the operator can dynamically respond to solr state changes. Until then, I'm not sure we are going to be able to fully support Solr as a first class Kubernetes offering.

Please let me know if I'm misunderstanding the sidecar method that y'all are proposing.

[swarupdonepudi]

@HoustonPutman makes total sense. While I agree that REST interface may not give the watch functionality that is offered by zkstatereader java version, I am trying to make it work using the same sidecar pattern by making the sidecar container offer a bi-directional grpc stream and the solr-operator can act as a client and get notified by the sidecar when there are changes.

While I also agree that implementing the watch functionality in golang is far more beneficial in the long than investing time work around it, I still think that there is value in reusing the java version since there is going to be more contributions to lucene-solr repo than this one at this point.

If you think that bidirectional grpc is a good idea, can you help me write a simple java codeblock to understand how to set up a watch on solrcollections in a standalone maven project by importing lucene-solr jar?

[swarupdonepudi]

Nevermind. I got the watch working in the java app. Now working on setting the directional stream.

[swarupdonepudi]

@HoustonPutman I was able to get the watcher prototype working as outlined in the diagram - https://github.com/bloomberg/solr-operator/issues/40#issuecomment-548977672

I created a bidirectional stream grpc server in java using springboot - https://github.com/swarupdonepudi/solr-zkstatereader-grpc-server

I also create a client in golang - https://github.com/swarupdonepudi/solr-zkstatereader-grpc-client

Both share the same proto file

https://github.com/swarupdonepudi/solr-zkstatereader-grpc/blob/master/src/main/proto/CollectionsWatcher.proto

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-client/blob/master/CollectionsWatcher.proto

Client connects to the grpc server running on localhost port 6565

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-client/blob/master/client.go#L13

Client then sends a request to the server to create and register a collections listener for a zookeeper

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-client/blob/master/client.go#L26-L32

Server creates a collections listener for the zkstatereader

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-server/blob/master/src/main/java/com/k8sctl/solrzkstatereadergrpc/CollectionsWatcherService.java#L59-L60

Server notifies the client when there is an update on collections

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-server/blob/master/src/main/java/com/k8sctl/solrzkstatereadergrpc/CollectionsWatcherService.java#L57

Client simply prints the old and new list of collections for now

https://github.com/swarupdonepudi/solr-zkstatereader-grpc-client/blob/master/client.go#L44-L60

[swarupdonepudi]

But @sepulworld asked me a really good question - how can this ability to setup a watch improve solr-operator?. I did not have an answer to this question.

I was hoping you could share your thoughts on the benefits of having the ability to setup watches on collections using zkstatereader.

[sepulworld]

I think the watch will make come in handy for autoscaling automation.

[HoustonPutman]

So sorry that I completely missed this earlier. I think it will help immensely for autoscaling automation. And generally let the operator automatically adapt to changes in the solr cluster state. I don't think it's exactly critical right now though, until we add better autoscaling stuff.

Looking through the stuff you provided, this looks cool. I'm not very knowledgable about GRPC, but I'll try to catch up quickly. The first thing I notice is that you don't have a license on your project. I would add the apache license real quick, so that there is no issue with us using it in this project.

If the stream/watching stuff is there. Then I think this will work well and I'd be all for using it.

Quick question, it looks like it from what I'm seeing but this will be able to handle connections to different zookeeper ensembles and chRoots, correct?

[swarupdonepudi]

Glad to hear positive feedback from both of you @HoustonPutman and @sepulworld. I will add the apache license. Yes, this broker accepts zookeeper url as grpc request inputs which are used to either create new connection or use an existing zk connection. But we can totally iterate on the design. I was just prototyping it and will start cleaning it up and turn it into a proper project now.