[Feature] Proxy logging improvements

[arthurwozniak]

Hi there!

I recently attempted to migrate from kamal version 1.9.1 to 2.2.2 and encountered a couple of issues that require some workaround configurations for kamal and rsyslog to maintain the original behavior.

1. Compliance with logging.driver

It seems that the container running kamal-proxy is not utilizing the specified logging.driver configuration (if set) and is instead writing logs to a file. This change is unexpected, as traefik adhered to the logging driver specified in the configuration.

Current Configuration:

logging:
  driver: syslog
  options:
    tag: '{{ index (split .Name "-") 1 }}'

2. Passing custom docker options to proxy

To address the logging issue, I attempted to set docker options using the boot_config command.

$ kamal proxy boot_config set --docker_options log-driver=syslog -d staging
$ kamal proxy reboot -d staging
Note: This may cause a brief outage on each host. Are you sure? [y, N] (N) y
...

However, I encountered an error:

INFO [e1b4c244] Running docker run --name kamal-proxy --network kamal --detach --restart unless-stopped --volume kamal-proxy-config:/home/kamal-proxy/.config/kamal-proxy $(cat .kamal/proxy/options || echo "--publish 80:80 --publish 443:443 --log-opt max-size=10m") basecamp/kamal-proxy:v0.8.1 on A.B.C.D
Releasing the deploy lock...
ERROR (SSHKit::Command::Failed): Exception while executing on host A.B.C.D: docker exit status: 125
docker stdout: Nothing written
docker stderr: docker: Error response from daemon: unknown log opt 'max-size' for syslog log driver.
See 'docker run --help'.

It appears that Kamal internally sets the max-size option, which conflicts with the syslog log driver. This can be resolved by manually editing .kamal/proxy/options on host to remove --log-opt max-size=10m.

3. Definition of unwanted log fields

Another useful feature would be the ability to specify fields to exclude from log entries. Currently, the log entries contain many fields that may not be necessary for certain use cases, such as:

{
  "time": "some_val",
  "level": "some_val",
  "msg": "some_val",
  "host": "some_val",
  "port": "some_val",
  "path": "some_val",
  "request_id": "some_val",
  "status": "some_val",
  "service": "some_val",
  "target": "some_val",
  "duration": "some_val",
  "method": "some_val",
  "req_content_length": "some_val",
  "req_content_type": "some_val",
  "resp_content_length": "some_val",
  "resp_content_type": "some_val",
  "client_addr": "some_val",
  "client_port": "some_val",
  "remote_addr": "some_val",
  "user_agent": "some_val",
  "proto": "some_val",
  "scheme": "some_val",
  "query": "some_val",
  "resp_x_runtime": "some_val"
}

Since we can already define proxy.logging.request_headers and proxy.logging.response_headers, I suggest adding a proxy.logging.dropped_fields array for this purpose.

Fields can be filtered out in the rsyslog configuration, but this introduces significant complexity to what is otherwise a straightforward tool.

However, this is likely a feature request for the kamal-proxy repo as logging is handled there.

Thank you for your hard work on this project! Keep rolling! 🚀 💪

[djmb]

In Kamal 1 we used the logging options from the app config for the proxy. But that's not something that we want to do anymore since Kamal 2 supports running multiple apps through one proxy so the apps could have conflicting settings.

If you are only deploying one app though to a server, that's not a concern and you can use kamal proxy boot_config set -y in a pre-deploy hook to force the servers to have the right settings.

RIght now though it will always try to set a max-size, so that's a bug since it's breaking things for the syslog driver.

[djmb]

#1152 will you allow you to remove the max size log opt, via kamal proxy boot_config set

[arthurwozniak]

@djmb Thanks! Your fix combined with pre-proxy-reboot hook will solve points Compliance with logging.driver and Passing custom docker options to proxy.

[emmceemoore]

For anyone else wanting to make changes to the kamal-proxy docker options I would just like to point out that running a kamal proxy boot_config set ... command in the pre-proxy-reboot only runs when you do a kamal proxy reboot (duh!).

When it came time for me to kamal setup another destination I was confused as to why kamal-proxy didn't have the desired settings, then I realized that I forgot to do a kamal proxy reboot. 🤦

For me, I found that running the kamal proxy boot_config set ... command from both the docker-setup hook and the pre-proxy-reboot hook made it easier for both existing and new destinations to have the desired configuration.