Expose mongo/redis url

[sonnd08]

Describe the bug Hi, I'm trying to expose a public domain for my MongoDB/Redis with VirtualServer but it does not work. Could you guys take a look to see what is problem here?

mongodb://user:password@my-mongo-svc:27017/testdb => this work (internally)

mongodb://user:[email protected]:80/testdb => Invalid message size: 1347703880, max allowed: 67108864

To Reproduce

apiVersion: v1
kind: Service
metadata:
  name: my-mongo-svc
  namespace: default
spec:
  ports:
    - port: 27017
      targetPort: 27017
      protocol: TCP
  selector:
    app: my-mongo

---

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: mongo-domain
spec:
  host: mongo.mydomain.com
  upstreams:
    - name: my-mongo
      service: my-mongo-svc
      port: 27017
  routes:
    - path: /
      action:
        pass: my-mongo

Update: May be related to this? https://stackoverflow.com/questions/31853755/how-to-setup-mongodb-behind-nginx-reverse-proxy

[github-actions[bot]]

Hi @sonnd08 thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[brianehlert]

You should be using TransportServer to expose TCP/UDP. The StackOverflow link mentions the NGINX stream module. https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/ VirtualServer is specific to HTTP/s traffic.

Also, be sure to define a port listener in the GlobalConfiguration resource: https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/#listener

This demonstrates using TransportServer and GlobalConfiguration: https://github.com/nginxinc/kubernetes-ingress/tree/v2.2.2/examples/custom-resources/basic-tcp-udp

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 10 days with no activity.

[nlsrchtr]

Hi @brianehlert, thanks for your explanation, this clarified a lot for me as well. But I was wondering how I could expose a service, which is not in the namespace of the nginx-controller? The transport server does only allow services in the same namespace afaik.

Thanks for your support!

[brianehlert]

https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/cross-namespace-configuration

[rahulmr]

@sonnd08 were you able to get a solution for this? I am looking exactly for such scenario, where my mongodb is in k8s namespace , I want access it using mongodb://user:[email protected]:27017 - how to use nginx ingress to do so?

[brianehlert]

This is commonly accomplished using L4 routing instead of L7 routing. @rahulmr have you taken a look at the TransportServer resource? (and be sure to not miss defining a listener using GlobalConfig and making sure the ports are exposed on your deployment / service) https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/

https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/basic-tcp-udp

[rahulmr]

Thanks @brianehlert , yes I saw the documentation on Transport Server, I will try to implement the same. My only worry was it should be dynamic. As I need to create multiple namespaces with mongodb in each. So it should have dns , not sure how external dns will help in this case. Means I will have to create a normal ingress as well as TransportServer both. So bit confused which ports I should take into account. I will update you or ask if I get an issue. And yes I already knew that this is L4 which is not supported by AWS ALB ingress controller which we are using currently.

[brianehlert]

Ah. So you are trying to publish different / multiple Redis instances, each with unique hostnames but with the same port. I don't know enough about the Redis client, is it an HTTP client? or is it TCP or gRPC?

What I think you are trying to avoid is mapping a unique port for each Redis instance, which is what doing this at L4 essentially forces.

[rahulmr]

@brianehlert thanks for replying back again. Let me explain a bit in detail.

I have say 3 namespaces based on city Say dev-la, dev-ny and dev-se I have deployed 3 services and 1 mongodb in it.

svc-1
svc-2
svc-3
mongodb

Since svc-1, svc-2 and svc-3 are web based ui and api services it is easy for me to have ing using the alb ingress controller so consider they are as below ingresses

svc-1-ing    svc1ing.la.example.com tgtgrp-address-public-alb-ingress-controller    => this is already done using ingress contoller
svc-2-ing    svc2ing.la.example.com tgtgrp-address-public-alb-ingress-controller    => this is already done using ingress contoller
svc-3-ing    svc3ing.la.example.com tgtgrp-address-public-alb-ingress-controller    => this is already done using ingress contoller
mongodb   mongodb.la.example.com some-other-load-balancer-supporting-tcp  => this is what I want to achieve

Same I need to do for dev-ny and dev-se. It was to easy to configure web and api services.
I was able to use kong ingress controller to expose mongodb by using TCPIngress but problem is redirection is not happening based on hostname which is same as domain name mongodb.la.example.com. All 3 domains are connecting to same DB. I am using external-dns and route53 for creating the domain entries (dns).

So basically I want ingress like mongodb.la.example.com should be able to connect to pod in dev-la namespace with the mongo db protocol using below command.
mongosh mongodb://user:[email protected] Similarly for other namespaces I should be able to do below.
mongosh mongodb://user:[email protected]
mongosh mongodb://user:[email protected]

I want to deploy this dynamically so that in future if there is a new namespace say dev-mumbai, dev-nagpur and dev-pune, then it should be done automatically through pipelines.
For mongodb I think it will work with TCP and upstream, it is not http,https or gRPC.

[brianehlert]

Here is the hitch. You can do this using TransportServer if you have a DNS that maps to a unique port for each mongo instance.

But VirtualServer and Ingress are matching the http host header. And unless the mongo client is http, that won't exist with a TCP connection. The TCP connection is at the physical layer. Each mongo instance would require a unique port.

Now, if you have control over DNS, this can totally be done. But not magically like VirtualServer does through external-dns by matching up the hostname automatically. You need to automate the DNS layer to map the DNS to the port forward on the ingress side. And each mongodb is a unique port listener on the ingress.

This can be done with TransportServer and adding a new listener with GlobalConfiguration and checking the open ports of your Service. Multi step, becuase it generally passes through the hands of multiple humans to expose additional ports.

[brianehlert]

All that said. I can see how what you are looking for is a connection between the TransportServer object, a way to define a hostname that external-dns could then hook on to, and lastly the ability to define new listener ports on the fly.

All those moving parts together could enable something like I think you are looking for.

There would need to be some types of security checks and balances of course.

[rahulmr]

@brianehlert you are right. Thanks. I see there is lot to do, so I was also checking some other solutions like Istio Gateway but I am not so sure about it. Second trick could be exposing the NodePort on load balancer. This will also need some coding. Actually, my best alternative was to provide a web admin ui like mongoku to developers who can connect using svc internal dns like mongodb://user:[email protected]:27017 Let's see which thing I will finally decide to do, as both will take time to implement. Mongoku solution is good enough. It allows edition of collections in easy way.

[jasonwilliams14]

@rahulmr That is definitely an option, if you create a web admin UI, which I believe would operate at Layer 7. As @brianehlert mentioned, your current setup would be all L4, and the only way to distinguish between your deployments would be to have them then listen on different ports. If you try the web admin UI, you can use our CRDs (virtualServer and virtualServerroute) to easily route traffic across different namespaces. HTH