Add Custom templates to enable user specified labels to generate Caddyfile blocks

[SvenDowideit]

@lucaslorentz I started to work out how to implement your proposal - initially for non-swarm containers

see https://github.com/SvenDowideit/caddy-docker-proxy/blob/cdp-macros-experimentation/plugin/generator/templates.go

annoyingly, compose-file config's are Swarm mode only - and one of my targets is hooking up docker run, so we need another way to mix in template definitions when not using Swarm.

I'm hoping to have it done before the end of September :)

copied from original issue discussion

examples of use:

• docker run -it --label virtual.port=80 --name nginx --network cirri_proxy nginx
• docker-compose
• docker swarm service

{{ if index labels "virtual.port" }}
*.loc.alho.st loc.alho.st {
			import dns_api_gandi
			@{{matcher}} {
					host {{hostname ((index labels "virtual.host"))}}.loc.alho.st
					{{ if index labels "virtual.path" }}
					path {{ index labels "virtual.path" }}
					{{ end }}
			}
			route @{{matcher}} {
					reverse_proxy {{upstreams ((index labels "virtual.port" | int)) }}
			}
}
{{ end }}

• [x] for Swarm mode, templates can be added using Swarm config's by using the label caddy.template (internally the template is identified using the config name, so it can be updated)
• [x] Templates need a non-swarm way to inject templates - the $XDG_CONFIG_DIR/caddy/docker-proxy/ is watched for file changes and are loaded as templates
• [x] For each container/service found in swarm cluster, all templates will be invoked with that docker object as input data. If a template output is a non empty/whitespace string, the output is converted to caddyfile and merged into the final generated caddyfile, along with other contents generated by the traditional label to caddyfile conversion.
• [x] It's template responsibility to decide if it should render or not a caddyfile for that docker object
• Templates would have the following functions available:
  • [x] sprig: we had that before, I think something accidentally dropped it
  • [x] upstreams: to easily generate upstreams addresses
  • [x] labels: a uniform way of retrieving all labels of current docker object (container or service)
  • [x] labels: add image labels
  • [x] entitytype: a way to identify the type of docker object the template is running against (container or service). This might be required in order to write templates compatible with both.
  • [x] hostName: a way to get container_name, service_name, or set from param (in this PR's example case, a label virtual.host)
• [x] documentation

[SvenDowideit]

one problem I just came across is what happens when a user starts with non-swarm with compose, then turns on swarm mode - the shared bridge mode network isn't used, and docker stack deploy creates another with the same name, but in overlaymode

may need to detect that there are 2 networks with the same name, and show an error to help users debug.

[lucaslorentz]

@SvenDowideit Glad you started working on it.

[SvenDowideit]

argh - will get back to it soon - need to mock up some form of persistent session-id :)

[SvenDowideit]

mmm, I wonder what the right place to watch for tmpl files is - https://caddyserver.com/docs/conventions#file-locations

for non-swarm, I think I'm going to need a volume dir - $XDG_CONFIG_HOME/caddy-docker-proxy/*.tmpl or $XDG_DATA_HOME/caddy-docker-proxy/*.tmpl ?

XDG_CONFIG_HOME=/config
XDG_DATA_HOME=/data

[francislavoie]

/config seems okay to use for that. Really the only thing Caddy uses it for currently is the autosave.json, which is to make the --resume option work for users who primarily use the API for configuration. You could put all your stuff in like /config/tmpl/ no problem 😄

[SvenDowideit]

I've just implemented support for virtual_1 .. virtual_9 for host, domanin, port, path - using just template code for a new production setup I need to get out the door in the next week.

I'll hopefully get back to writing docs and examples soon

[SvenDowideit]

@lucaslorentz I'm basically done - this is in use at my (about to not be) workplace, and they'll be continuing its use - and I kinda think i'll use it for my new gig too, but I havn't started that yet :)

ah, ok, some tests to fix

[SvenDowideit]

and yup, simple enough thing to fix, I'll get to it soon

[lucaslorentz]

Yes. I have fixed it by changing some test consts into dynamic variables and using caddy path to compute expected log output. But I wasn't able to push it into your PR. So I will wait for your fix.

[SvenDowideit]

@lucaslorentz mmm, that last commit fixes that, but then we fail in running the integration tests - I'm not sure how those work

[lucaslorentz]

Thanks, @SvenDowideit, I re-triggered, could be some flaky integration test Will check again later

[SvenDowideit]

@lucaslorentz I don't work where I built this anymore, but they do seem to still use it, and I'm thinking of using it again - can we get this to a state were you're happy to merge it?

[SvenDowideit]

TODO: I think I hit a cgroupsv2 issue on my rarely used debian bullesys box

{"level":"error","ts":1638237048.64944,"logger":"docker-proxy","msg":"Failed to get ingress networks","error":"Cannot find container id in cgroups: 0::/\n"}
{"level":"warn","ts":1638237048.6551552,"logger":"docker-proxy","msg":"Container is not in same network as caddy","container":"/blog_wordpress_1","container id":"b88db68534622965b05ad0dccb60ffabc14dd26f6eb04bcf385f543dcb4d1bee"}
{"level":"warn","ts":1638237048.6567273,"logger":"docker-proxy","msg":"Container is not in same network as caddy","container":"/test","container id":"427dfb025a1f1d50c8e84c00d0830deef880de88e77f173832b2272f1b4b91c3"}
{"level":"error","ts":1638237053.835355,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_addr":"10.13.13.127:57662","proto":"HTTP/2.0","method":"GET","host":"blog_wordpress_1.mini.alho.st","uri":"/","headers":{"User-Agent":["curl/7.74.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"blog_wordpress_1.mini.alho.st"}},"duration":0.000259741,"status":502,"err_id":"y4fejbtri","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}

[lucaslorentz]

Hey @SvenDowideit I'm not planning to merge this one because so far everything indicates that there is not enough audience for the template feature. Although you did a great job, it adds some maintenance overhead to the project. I think for now the best is that you maintain your fork with this feature, sorry for not signaling this earlier. I will leave this PR open for some time so maybe more people can +1 this feature.

[SvenDowideit]

coolieo - I have plans for something like this for a Cryptoworkbench app, but it'll be a few months before we work out the right approach too.

[lucaslorentz]

Closing this for now. @SvenDowideit I wonder if caddy-gen isn't more suitable for your use case. Check this comparison I wrote: https://github.com/lucaslorentz/caddy-docker-proxy/issues/388#issuecomment-1207283624

I wasn't aware of it before, but seems to be exactly how you would like to manage caddy reverse proxies.

[SvenDowideit]

yeah, seems reasonable, i'm not using this code, as i've moved on to other $workplaces, and I'm not 100% sure if this is being used anymore.