Kubernetes Recommendations

[TheNotary]

Hello, I'm shopping out a self-hosted email server, and this project looks very appealing. Does anyone have a helm chart or kubernetes manifest files and a config they're happy with for deploying this to their k8s cluster?

[mjl-]

hi @TheNotary! i don't know of any kubernetes-supporting configs for mox. i'm not running mox in a docker container myself. i think a starting point would be the docker-compose file and its instructions, see https://github.com/mjl-/mox/blob/main/docker-compose.yml.

[brandonros]

the only thing that stands out to me is the need for host networking. I definitely get why, just thinking how that will translate through to cluster/pods/CNI/all that.

providing the host IP through an env var or something clever could be an option, but the rest of this comment makes me nervous/concerned it won't work

https://github.com/mjl-/mox/blob/main/docker-compose.yml#L37C5-L38C71

[mjl-]

The comment about needing an external IP is a bit too strong. It isn't strictly needed, but it normally makes a mox installation simpler when there's no NAT or anything in between. Running the quickstart with "-hostname mx.example.com" will generate a config that uses the IPs of mx.example.com in its config, and if the actual available IPs to bind to are all non-public, the quickstart will assume the public IPs (of hostname) are NATed to the private IPs and generate a config that listens on the private IPs and marks the public IPs as "NATIPs" (which are used for generating DNS SPF records and such).

Do kubernetes containers normally get access to public IPs? Or do containers always have private IPs with kubernetes setting up NAT/forwarding rules to direct incoming external traffic to the container?

I'm also wondering what the flow typically is for containers that need configuration. For a mox setup, the admin still has to create DNS records initially. I have a part of a prototype where mox can start with a guided web-based setup. When that is ready, it probably makes sense to automatically start that if there is no mox config file yet.

[TheNotary]

Do kubernetes containers normally get access to public IPs?

In k8s, we refer to the process being deployed (e.g. /bin/mox) as a container running in a pod resource. The process (pod) must be explicitly granted inbound network connectivity through a service resource. To give a process a specific IP address, a loadbalancer type service can be used.

(just_the_mox_service_resource.yaml)

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    # Cloud Specific Section:
    # "Resource group" for the Azure Kubernetes load balancer
    service.beta.kubernetes.io/azure-load-balancer-resource-group: my-resource-group-id
    # Name/id of the pre-allocated cloud native IP to attach the service to
    service.beta.kubernetes.io/azure-pip-name: my-mail-ip
spec:
  type: LoadBalancer
  ports:
  - port: 80
  - port: 443
  - port: 995 # are these the right ports?
  - port: 110
  - port: 143
  selector:
    app: my-mox-pod

To apply the above configuration to a k8s deployment, you would use the below command:

kubectl apply -f just_the_mox_service_resource.yaml

Of course we would need to create a pod resouce as well which would be another yaml that specified the container image to use, and associations with environment values through yet another kubernetes resource type, configmap resource.

Finally there's a pvc resource that needs to be defined which is basically just a docker-compose volume and allows the process to read/write against a persistent volume.

For a mox setup, the admin still has to create DNS records initially I currently automate/ manage my DNS using a hand-written terraform repo, though I believe k8s can hook into cloud platforms and actually manage DNS from k8s deployment yaml files.

I think to make k8s deployments approachable, just the pod, configmap, pvc and perhaps a more generic type of service resource would be all that's required. Instead of using a cloud-native service resource, maybe it would be easier to specify either a nodeport or an ingress resource? The conversation here makes me think this is doable. I can give it a shot and share my work here if I wind up whipping something functional.

[thirdeyenick]

I am running all my services in a k8s cluster and I'd love to run mox in it as well. I am using a proxy in front of my k8s cluster which allows to speak the PROXY protocol. There is already an open issue for PROXY protocol support, which (imho) would make it possible to gather knowledge about the real client IPs, even when running in a pod.

[RobSlgm]

@TheNotary You would also need port: 25 (smtp). 465 (smtps) and 993 (imaps). 995 is only required if you support POP. Port 25 could pose a challenge with a deployment on azure (it's in my understanding blocked in most cases).

Another issue is a stable external IP for outbound traffic.

Would appreciate to see a working k8s setup.

[mjl-]

PROXY protocol supports seems good to have. It's been discussed before. It shouldn't be too hard to make a Go net.Listener that transparently deals with it, returning the appropriate addresses for RemoteAddr() calls on the net.Conn's it returns.

Would indeed be good to have a working kubernetes examples.