nats chart should support setting the service.spec.type

[benmathews]

The nats chart service template doesn't specify service.spec.type. Therefore, it defaults to ClusterIP. To expose the cluster to outside, I would like to have NodePort as an option.

[wallyqs]

Would it be good if the Helm charts are extended to create a NodePort via a toggle. Here is an example to create one out of band as well:

apiVersion: v1
kind: Service
metadata:
  name: nats-nodeport
  labels:
    app: nats
  annotations:
    external-dns.alpha.kubernetes.io/hostname: sfo.nats.chat
spec:
  type: NodePort
  selector:
    app: nats
  externalTrafficPolicy: Local
  ports:
  - name: client
    port: 4222
    nodePort: 30222
    targetPort: 4222
  - name: websocket
    port: 443
    nodePort: 30223
    targetPort: 443

[jknezevic]

Is there any action on this? We would need this, as the requirement is that NATS is available inside the whole VPC via ILB, and not only in K8s cluster.

I've tried changing the values in the existing helm repo, but when I change the service type to LoadBalancer (with proper annotations), the NATS stateful sets are trying to establish routing. After that, as the stateful sets are unhealthy, jetstream can't establish contact with the meta leader, and the whole workload is in the restarting loop.

We are running inside GCP.

Modified service.yaml from charts/nats/templates/service.yaml:

spec:
  type: {{ .Values.service.type }}

Using the following values:

nats:
  jetstream:
    enabled: true
    fileStorage:
      enabled: true

cluster:
  enabled: true
  replicas: 3
  
metrics:
  enabled: true

serviceAnnotations:
  networking.gke.io/load-balancer-type: Internal

service:
  type: LoadBalancer

Logs from one of the pods (others are the same)

2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867290 [INF] Starting nats-server
2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867383 [INF] Version: 2.9.0
2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867387 [INF] Git: [517d9b3]
2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867421 [INF] Cluster: nats
2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867426 [INF] Name: nats-mq-1
2022-09-30 15:15:37.867 CEST[7] 2022/09/30 13:15:37.867466 [INF] Using configuration file: /etc/nats-config/nats.conf
2022-09-30 15:15:37.868 CEST[7] 2022/09/30 13:15:37.868571 [INF] Starting http monitor on 0.0.0.0:8222
2022-09-30 15:15:37.868 CEST[7] 2022/09/30 13:15:37.868642 [INF] Starting JetStream
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870336 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870353 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870356 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870359 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870362 [INF]
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870365 [INF] https://docs.nats.io/jetstream
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870367 [INF]
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870372 [INF] ---------------- JETSTREAM ----------------
2022-09-30 15:15:37.870 CEST[7] 2022/09/30 13:15:37.870383 [INF] Max Memory: 1.00 GB
2022-09-30 15:15:37.872 CEST[7] 2022/09/30 13:15:37.872405 [INF] Cluster name is nats
2022-09-30 15:15:37.872 CEST[7] 2022/09/30 13:15:37.872470 [INF] Listening for route connections on 0.0.0.0:6222
2022-09-30 15:15:37.890 CEST[7] 2022/09/30 13:15:37.890784 [ERR] Error trying to connect to route (attempt 1): lookup for host "nats-mq-2.nats-mq.nats.svc.cluster.local": lookup nats-mq-2.nats-mq.nats.svc.cluster.local on 10.16.0.10:53: no such host
2022-09-30 15:15:37.891 CEST[7] 2022/09/30 13:15:37.890890 [ERR] Error trying to connect to route (attempt 1): lookup for host "nats-mq-0.nats-mq.nats.svc.cluster.local": lookup nats-mq-0.nats-mq.nats.svc.cluster.local on 10.16.0.10:53: no such host
2022-09-30 15:15:37.957 CEST2022/09/30 13:15:37 Starting NATS Server Reloader v0.7.2
2022-09-30 15:15:37.957 CEST2022/09/30 13:15:37 Live, ready to kick pid 7 (live, from 7 spec) based on any of 1 files
2022-09-30 15:15:37.973 CEST[7] 2022/09/30 13:15:37.973317 [WRN] Waiting for routing to be established...
2022-09-30 15:15:38.058 CEST[35] 2022/09/30 13:15:38.058396 [INF] Prometheus exporter listening at http://0.0.0.0:7777/metrics
2022-09-30 15:15:39.881 CEST[7] 2022/09/30 13:15:39.881623 [WRN] Waiting for routing to be established...
2022-09-30 15:15:41.888 CEST[7] 2022/09/30 13:15:41.888772 [WRN] Waiting for routing to be established...
2022-09-30 15:15:43.896 CEST[7] 2022/09/30 13:15:43.896762 [WRN] Waiting for routing to be established...
2022-09-30 15:15:45.905 CEST[7] 2022/09/30 13:15:45.905337 [WRN] Waiting for routing to be established...
2022-09-30 15:15:47.359 CEST[7] 2022/09/30 13:15:47.359279 [WRN] Healthcheck failed: "JetStream has not established contact with a meta leader"
2022-09-30 15:15:47.913 CEST[7] 2022/09/30 13:15:47.913545 [WRN] Waiting for routing to be established...
2022-09-30 15:15:49.921 CEST[7] 2022/09/30 13:15:49.921302 [WRN] Waiting for routing to be established...
2022-09-30 15:15:51.929 CEST[7] 2022/09/30 13:15:51.929802 [WRN] Waiting for routing to be established...
2022-09-30 15:15:53.938 CEST[7] 2022/09/30 13:15:53.937914 [WRN] Waiting for routing to be established...
2022-09-30 15:15:55.946 CEST[7] 2022/09/30 13:15:55.946149 [WRN] Waiting for routing to be established...
2022-09-30 15:15:57.359 CEST[7] 2022/09/30 13:15:57.359761 [WRN] Healthcheck failed: "JetStream has not established contact with a meta leader"

Worth noting: everything works as expected when I don't mess up service. Also, creating everything with default options for service and then deleting existing service and creating the LoadBalancer service does not work (the same error occurs).

[caleblloyd]

The service type in the chart has to be a headless service, it is a requirement of the Stateful Set.

There are so many options for Services, WebSocket Ingresses, Service Meshes, etc these days that it is difficult to capture all of the resources that can be used to expose NATS in this chart. I have documented our recommended approach of using the NATS chart as a Dependency and then layering on the resources that are needed to expose NATS in #581

[stefanmcshane]

Whilst I understand that any helm chart can be imported as another helm chart, the usefulness of this chart is heavily, heavily reduced by not supporting the various standard deployment options, that most other tooling in the same category supports. Using a helm chart maintained by a company, and expecting the users to write their own which wraps it, are two very different use-cases.

[caleblloyd]

I agree but helm's lack of compasability is limiting.... The chart already must ship the headless service due to the Stateful Set. This service name is already the chart name and is immutable due to the Stateful Set.

What would the 2nd service be called? Would it be enabled by default? What options in the services would be editable? What ports would be set on it? What overrides would be provided? Would we disable client advertisements in NATS if this service were enabled?

It seems like a good problem to tackle in a major version bump. But if there were an elegant way to fit it into the existing chart it could be considered.