hashicorp
disabling enable_syslog does not mute syslog
Overview of the Issue
I can see that every consul agent puts log into syslog and sometimes its getting overwhelming. there is a configuration file option called enable_syslog which can be true or false. tried both values and i don't see any difference
Reproduction Steps
Steps to reproduce this issue, eg:
i watched /var/log/messages when starting consul agent with enable_syslog = true and enable_syslog = fale
Consul info for both Client and Server
Client info
agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = 8b4a3d95 version = 1.7.3 consul: acl = disabled known_servers = 3 server = false runtime: arch = amd64 cpu_count = 16 goroutines = 46 max_procs = 16 os = linux version = go1.13.7 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 32 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 138 members = 27 query_queue = 0 query_time = 1
Server info
agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = 9ea1a204 version = 1.7.2 consul: acl = disabled bootstrap = false known_datacenters = 1 leader = true leader_addr = 192.168.xx.xx:8300 server = true raft: applied_index = 1233099 commit_index = 1233099 fsm_pending = 0 last_contact = 0 last_log_index = 1233099 last_log_term = 324 last_snapshot_index = 1229138 last_snapshot_term = 301 latest_configuration = [{Suffrage:Voter ID:9fe763c0-bc2a-c600-ed8f-a4c33ed7cdd9 Address:192.168.xx.xx:8300} {Suffrage:Voter ID:ea4bdb5c-3c8f-2211-9283-774287f958a3 Address:192.168.xx.xx:8300} {Suffrage:Voter ID:2e62c57d-1d14-e898-3a00-5a445dbedacf Address:192.168.xx.xx:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 324 runtime: arch = amd64 cpu_count = 2 goroutines = 185 max_procs = 2 os = linux version = go1.13.7 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 32 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 138 members = 27 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 19 members = 3 query_queue = 0 query_time = 1
Operating system and Environment details
server node runs on Ubuntu Server 18.04 agent node runs on CentOS 7.7 both are amd64 architecture
Log Fragments
Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the gist of the log instead of posting it in the issue. Use -log-level=TRACE on the client and server to capture the maximum log detail.
I found that even though I defined log_file = /data/consul/log/consul.log in my configuration, the log was still output to /var/log/message. how to disable logging to /var/log/message?
Hi! My agent has additional lines in configuration: log_file = "/var/log/consul/" enable_syslog = false I see that /var/log/consul/ has now logs, but syslog still receives the same info and i can see it from /var/log/messages. I guess it's a bug that syslog output is not disabled when i have this configuration enabled. I'm hoping it will be fixed some day :)
As an alternative you could maybe configure your rsyslog not to put consul logs to /var/log/messages. Generally /etc/rsyslog.conf has /var/log/messages rule defined. You probably could tune the default rule to leave out consul. As a test i created /etc/rsyslog.d/consul.conf file and added one line to it: if $programname == 'consul' then /var/log/consul/consul.service.log with this configuration, one more copy of conslu log is generated :) maybe you could make some logical expression to leave consul logs out from /var/log/messages. For me, this is not a solution, because i have to manage multiple servers and manually reconfiguring default settings is not the way to go. journalctl still shows consul messages as you just hide it from /var/log/messages output
I added the following to my consul config:
"log_file" = "/var/log/consul/" "log_rotate_duration" = "24h" "log_rotate_max_files" = -1
then ensure consul owned /var/log/consul/ and it existed.
then I created /etc/rsyslog/consul.conf and added the following to ignore logs in syslog/messages:
if $programname == 'consul' then stop
then restarted rsyslog:
service rsyslog restart
Using salt to manage changes on servers made this easily to create a salt state to apply this across a fleet of servers.
Thank you for reporting this issue! I wonder if the syslog messages are coming from the init system and not directly from Consul. Maybe the init script is sending Consul stdout to syslog?
It seems like in Consul, at least in recent versions, this setting should be respected: https://github.com/hashicorp/consul/blob/v1.11.2/logging/logger.go#L74-L91
There's one more way Consul messages can end up in your syslog or journal file: Consul seems to write these to stdout/stderr, which is then being intercepted by systemd and written to the system journal. Here's the relevant answer on the Community Support forums.
In short, just add these two lines to the /usr/lib/systemd/system/consul.service (or wherever your unit files are):
StandardOutput=null
StandardError=null
