Multiple traefik ports

[stepbeekio]

Hey,

I'd love to have a way to configure traefik to listen on multiple ports for multiple entrypoints to enable services that might listen for other kinds of traffic - e.g. a port open for HTTP traffic and another for TCP traffic. My current thinking is that this would be an escape hatch from the default configuration so mrsk doesn't need to support this as packaged up feature.

Would something like this:

traefik:
  host_port: 8080
  additional_ports:
     - 9000
  entrypoints.mytcp.address: ':9000'

be of interest?

Alternatively, maybe it could look like this:

traefik:
  host_port: 8080
  additional_entrypoints:
    myentrypointname: ':9000'

producing a traefik arg like --entrypoints.myentrypointname.address=:9000.

This would also enable the traefik admin dashboard for interested users.

[stepbeekio]

I've done a little work on this locally. Using the additional_ports suggestion above, I've ended up with a config yaml that includes a lot of excess effort. I've omitted anything not strictly related.

service: logline

labels:
  traefik.tcp.routers.logstash.rule: 'HostSNI(`*`)'
  traefik.tcp.routers.logstash.entrypoints: logstash
  traefik.tcp.services.logstash.loadbalancer.server.port: 10516
  traefik.http.routers.logline.entrypoints: web
  traefik.http.services.logline.loadbalancer.server.port: 8080

traefik:
  additional_ports:
    - 10516
  args:
    'entrypoints.web.address=:80': true
    'entrypoints.logstash.address=:10516': true
    accesslog: true

I have a PR ready to go if this is an acceptable UX for now - or am interested in hearing about how 37s feels about the tradeoff between deeper traefik integration vs an escape hatch like this @dhh.

I'm particularly interested in this use-case as a motivating feature for us to move from platforms like Heroku et al where the ability to open a plain ol' TCP port isn't offered. The gap between PaaS and managed K8S or Nomad etc. is pretty sparse so I'd love to have mrsk fill this for us.

[stepbeekio]

Opened a PR.

[stepbeekio]

Updated PR in response to feedback from @kjellberg. This is the final result:

labels:
  traefik.tcp.routers.logstash.rule: 'HostSNI(`*`)'
  traefik.tcp.routers.logstash.entrypoints: logstash
  traefik.tcp.services.logstash.loadbalancer.server.port: 10516
  traefik.http.routers.logline.entrypoints: web
  traefik.http.services.logline.loadbalancer.server.port: 8080

traefik:
  ports:
    - 80
    - 10516
  args:
    'entrypoints.web.address=:80': true
    'entrypoints.logstash.address=:10516': true

I'm fully overriding the ports published by traefik to avoid any confusion around how additional ports would have behaved with traefik defaults.

[kjellberg]

Thanks, really looking forward to be able to customize ports! But.. an entrypoint will always require a port and vice versa right? Maybe just using entrypoints instead of ports would be even better 😆

traefik:
  entrypoints:
    web: 80
    dashboard: 8080

Your example could then look like something this:

...
  labels:
    traefik.tcp.routers.logstash.rule: 'HostSNI(`*`)'
    traefik.tcp.routers.logstash.entrypoints: logstash
    traefik.tcp.services.logstash.loadbalancer.server.port: 10516
    traefik.http.routers.logline.entrypoints: web
    traefik.http.services.logline.loadbalancer.server.port: 8080

traefik:
  entrypoints:
    web: 80
    logstash: 10516

And the user can focus on configuring traefik instead of docker. I would love to see this extended to work for volumes as well. Which would make the following possible and enable configuration for things like ssl certificates with letsencrypt:

traefik:
 entrypoints:
   web: 80
   websecure: 443
 volumes:
   - /letsencrypt/acme.json:/letsencrypt/acme.json

But let's get some feedback from @dhh first.

[nerdinand]

I'm very interested in letsencrypt-support. That's the only thing keeping me from using mrsk right now. Let me know if I can help somehow!

[dhh]

Hmm, I don't love marrying MRSK closer to Traefik. I'd rather see if we can't find a way to support this without needing new, Traefik specific configuration options that we translate. So that means doing this configuration in a generic way, like we do by allowing label specifications, or args to the starting command.

[kjellberg]

Ok, understood! Not sure though if you're referring to the whole issue or just my proposal re entrypoints. But the thing is, there's no way (what I know, please correct me if I'm wrong cause I really need this 😁 ) to set docker arguments right now (for example additional ports and volumes). If you don't like idea of ports and volumes, we at least need a way to configure Docker args.

  def run
    docker :run, "--name traefik",
      "--detach",
      "--restart", "unless-stopped",
      "--log-opt", "max-size=#{MAX_LOG_SIZE}",
      "--publish", port,
      "--volume", "/var/run/docker.sock:/var/run/docker.sock",
      # <------  Docker args (--publish & --volume) must be inserted here before "traefik"
      "traefik",
      "--providers.docker",
      "--log.level=DEBUG",
      *cmd_option_args # <--- These are traefik specific args (--api.dashboard). Cant set port or mount volumes here.
  end

But in my opinion, configuration will still not be generic anyways since mrsk users then will have to learn traefik configuration. Most deploy.yaml files will consist of 99% traefik labels :')

Having a small API for basic stuff like "domain", "ports" and "volumes" makes it really easy to get started. And user configurations can stay the same even though MRSK one day decides to switch to Caddy or something else.

[dhh]

Yeah, it's exactly that we should try to stay abstracted such that we could change "Traefik" to "caddy" or anything else, and the options would all stay the same. Seems to me that if we added options, like we just did for app servers, we'd be able to get publish and volume in there, yeah?

[stepbeekio]

Hmm, I don't love marrying MRSK closer to Traefik. I'd rather see if we can't find a way to support this without needing new, Traefik specific configuration options that we translate.

@dhh @kjellberg That is the motivation around only adding the ports config. The concession to UX is that a user that self-selects into manual port management would need to also pass additional args to configure traefik entry points.

My main concern is that an unwitting user could end up manually overriding ports, mess up the default traefik config and have a bad time as a result.

[stepbeekio]

@kjellberg Would we want to fully override the dockers args if the user defined any? In your example, would overriding the restart policy work? Would restricting to ports and volumes be easier than resolving arbitrary args?

[dhh]

I think if you're down this track, you're already off-piste. Better know your way around the terrain.

I'd be happy to see all the options be defaults you can overwrite.

[stepbeekio]

@dhh @kjellberg Would it be better to have a config like this:

traefik:
  ports:
    - 80
    - 10516
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./letsencrypt/acme.json:/letsencrypt/acme.json
  restart: unless-stopped
  log-opt: '...' 
  image: traefik
  ... # other options
  args:
    'entrypoints.web.address=:80': true
    'entrypoints.logstash.address=:10516': true

Or to acknowledge that users might want to run traefik in a way that mrsk doesn't know about (or want to know about) and to have something like this:

traefik:
  cmd: "docker run --detach --name traefik --restart unless-stopped --publish 80..."

Which allows a user to do whatever weird and wonderful stuff they like by fully deploying the escape hatch. From your example @dhh, a user might even write something like:

traefik:
  cmd: "docker run --name traefik --detach --publish 80 -e VIRTUAL_HOST=api.example.com jwilder/nginx-proxy

Happy to do the work around this, but I'm not sure which option is right for mrsk.

[dhh]

I'd like to see it as:

traefik:
  options:
    publish:
      - 80
      - 10516
    volumes:
      - ...
  args:
    entrypoints.web.address: :80

[stepbeekio]

@dhh Updated https://github.com/mrsked/mrsk/pull/100 to support this config:

traefik:
  options:
    publish:
      - 80
      - 10516
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  args:
    entrypoints.web.address: ':80'

And deployed to a production environment with it. @kjellberg We're using TLS termination on the Digital Ocean load balancer, so Let's Encrypt support is out of my wheelhouse.

[kjellberg]

Nice, but please allow options to take any option, not only publish and volumes. You can have a look at how it was done with servers in PR https://github.com/mrsked/mrsk/pull/95. I guess you can just copy #cmd_option_args, but modify it a bit to accept arrays (for multiple ports etc):

  def run
    docker :run, "--name traefik",
      "--detach",
      "--restart", "unless-stopped",
      "--log-opt", "max-size=#{MAX_LOG_SIZE}",
      "--publish", port,
      "--volume", "/var/run/docker.sock:/var/run/docker.sock",
      *option_args, # <---options goes here here
      "traefik",
      "--providers.docker",
      "--log.level=DEBUG",
      *cmd_option_args # <- maybe rename this to something with traefik?
  end

We're using TLS termination on the Digital Ocean load balancer, so Let's Encrypt support is out of my wheelhouse.

Thanks, I will try with that, but I read something about that it requires us to switch to DO's nameservers. I want to use Cloudflare with Full SSL support. The Flexible SSL they offer doesn't work very well for me with Rails and CSRF - or does someone here have a hack?

[stepbeekio]

Updated PR - sorry, I misunderstood the desired config.

[stepbeekio]

@kjellberg would you like to move let's encrypt chat to another ticket? I'm happy to be involved but won't use it in prod so don't think I'm the right dev for it.

[kjellberg]

@kjellberg would you like to move let's encrypt chat to another ticket? I'm happy to be involved but won't use it in prod so don't think I'm the right dev for it.

I'd love to continue chat about different solutions in https://github.com/mrsked/mrsk/discussions/112. If your PR get's merged, we only have one small step left: To create an empty acme.json file on the host server(s) with correct permissions. Not sure if this is within the scope of MRSK or if it's something we'll have to do manually.