Solr Cloud only allows 1 exposed port.

[RichieHowell]

It would be helpful to be able to supply a list of ports in the solrAddressability instead of only have a single podPort possible. My specific use case is to be able to have both the normal port exposed to access to the solr cloud members but also the jmx port. My current workaround is to sidecar in a socat container to expose this port. I see no way to really have JMX function without having multiple exposed ports, and datadog monitoring uses JMX. I am happy to supply any more details necessary to understand this use case in full.

[HoustonPutman]

So would you need anything else beyond just having an additional list of ports in the customPodOptions?

This should be pretty straightforward to add.

[HoustonPutman]

Also I guess that there need to be options to allow exposing the ports via the services and ingresses...

[HoustonPutman]

Removing this from the v0.4.0 timeline due to a lack of information.

We can schedule it for v0.5.0 if more information on the use case, or a PR is provided.

[fernaspiazu]

Hi, any news about the possibility to open more ports in the solr container for JMX? Thank you in advance

[HoustonPutman]

There is more information needed for the request to implement it. Basically do the ports merely need to be open on the pod, or do they need to be available via the various services as well?

[plumdog]

@HoustonPutman I think that in the specific case (Datadog monitoring, which I'm interested in too) just exposing the ports on the pod would be sufficient. Maybe someone has a need to do more complicated things to expose the port in a service or ingress eventually idk, but exposing the port on the pod would be a very useful addition.

[plumdog]

I now think this change is not actually needed to be able to gather JMX metrics. I happened to read the following while doing something completely different, from kubectl explain.pod.spec.containers.ports:

Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network.

So I tried just setting some JMX options and pointing Datadog at the port. My changes were (broadly) as follows:

In the values passed to the Datadog chart:

agents:
  image:
    tagSuffix: jmx

And then in my SolrCloud:

apiVersion: solr.bloomberg.com/v1beta1
kind: SolrCloud
# ...
spec:
  customSolrKubeOptions:
    podOptions:
      annotations:
        ad.datadoghq.com/solrcloud-node.check_names: '["solr"]'
        ad.datadoghq.com/solrcloud-node.init_configs: '{"is_jmx": true, "collect_default_metrics": true}'
        ad.datadoghq.com/solrcloud-node.instances: '{"host": "%%host%%", "port": 18983}'
  solrOpts: >-
    # ...
    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.local.only=false
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.port=18983
    -Dcom.sun.management.jmxremote.rmi.port=18983

And then I started getting the metrics from https://github.com/DataDog/integrations-core/blob/master/solr/metadata.csv arriving in Datadog (some took a while to come through, not sure if that's because Datadog or because Solr).

For reference, I took the Solr options from https://solr.apache.org/guide/8_9/using-jmx-with-solr.html#configuring-a-remote-connection-to-solr-jmx and the Datadog options from https://docs.datadoghq.com/agent/guide/autodiscovery-with-jmx/?tab=containeragent#autodiscovery-annotations

Some things my config above doesn't do that might be a good idea:

• use authentication when connecting to JMX from Datadog. In my case this isn't really a concern, but I think is easy, so I might as well. I can imagine situations where the JMX options I'm setting above would be bad.
• set the pod name as a tag for Datadog, which seems to be missing, so I'm just getting instance:solr-[podIP]-18983 which isn't lovely
• I've not looked at getting anything from Zookeeper.

(The above is quite specific to Datadog, but I think shows that this issue doesn't block gathering of JMX metrics from pods. I hope it is a useful reference.)