Broker Error Log: feeder: error occured while processing client

[itsul]

Hi,

one of our customer has sometimes problems with a outside poller connection. Sometimes the status icon from the display counter is in a red/error state. While the problem was present the vpn was working fine and a test telnet connection to bbdo port 5669 was successfully. Only the broker itself was disconnected. We investigated much time in troubleshooting and increased the log. The monitoring system is the only system with problems, other applications are working well via VPN include important applications like SAP ( user will report immediately if an error occured).

Display Poller Broker Logsettings:

Centreon Versions: centreon-broker-cbmod-3.0.8-1.el6.x86_64 centreon-broker-storage-3.0.8-1.el6.x86_64 centreon-syslog-frontend-1.5.4-1.el6.noarch centreon-engine-1.7.2-3.el6.x86_64 centreon-connector-1.1.2-1.el6.x86_64 centreon-common-2.8.12-7.el6.noarch centreon-bam-server-3.5.5-1.el6.noarch centreon-bi-server-3.1.5-1.el6.noarch centreon-broker-3.0.8-1.el6.x86_64 centreon-connector-ssh-1.1.2-1.el6.x86_64 centreon-perl-libs-2.8.12-7.el6.noarch centreon-poller-centreon-engine-2.8.12-7.el6.noarch centreon-pp-manager-2.2.0-4.el6.noarch centreon-2.8.12-7.el6.noarch centreon-engine-daemon-1.7.2-3.el6.x86_64 centreon-connector-perl-1.1.2-1.el6.x86_64 centreon-release-3.4-4.el6.noarch centreon-base-config-centreon-engine-2.8.12-7.el6.noarch centreon-syslog-server-1.2.5-6.el6.noarch centreon-engine-extcommands-1.7.2-3.el6.x86_64 centreon-broker-cbd-3.0.8-1.el6.x86_64 centreon-clib-1.4.2-1.el6.x86_64 centreon-broker-core-3.0.8-1.el6.x86_64 centreon-trap-2.8.12-7.el6.noarch centreon-web-2.8.12-7.el6.noarch centreon-license-manager-1.0.1-3.el6.noarch

OS: CentOS 6.9

Additional environment details :Hardware maschine with a central and a display poller. Display poller is located in the US and connected via MPLS from Telekom.

Broker log from display poller with a human readable dateformat: [Thu Sep 28 06:23:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13866': TCP peer '1.2.3.4:37084' is disconnected [Thu Sep 28 06:28:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13867': TCP peer '1.2.3.4:38130' is disconnected [Thu Sep 28 06:33:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13868': TCP peer '1.2.3.4:39174' is disconnected [Thu Sep 28 06:38:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13869': TCP peer '1.2.3.4:40220' is disconnected [Thu Sep 28 06:43:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13870': TCP peer '1.2.3.4:41272' is disconnected [Thu Sep 28 06:48:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13871': TCP peer '1.2.3.4:42326' is disconnected [Thu Sep 28 06:53:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13872': TCP peer '1.2.3.4:43376' is disconnected [Thu Sep 28 06:58:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13873': TCP peer '1.2.3.4:44426' is disconnected [Thu Sep 28 07:03:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13874': TCP peer '1.2.3.4:45478' is disconnected [Thu Sep 28 07:08:10 2017] error: feeder: error occured while processing client 'central-broker-master-input-13875': TCP peer '1.2.3.4:46524' is disconnected [Thu Sep 28 07:13:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13876': TCP peer '1.2.3.4:47568' is disconnected [Thu Sep 28 07:18:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13877': TCP peer '1.2.3.4:48616' is disconnected [Thu Sep 28 07:23:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13878': TCP peer '1.2.3.4:49668' is disconnected [Thu Sep 28 07:28:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13879': TCP peer '1.2.3.4:50714' is disconnected [Thu Sep 28 07:33:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13880': TCP peer '1.2.3.4:51766' is disconnected [Thu Sep 28 07:38:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13881': TCP peer '1.2.3.4:52812' is disconnected [Thu Sep 28 07:43:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13882': TCP peer '1.2.3.4:53864' is disconnected [Thu Sep 28 07:48:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13883': TCP peer '1.2.3.4:54916' is disconnected [Thu Sep 28 07:53:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13884': TCP peer '1.2.3.4:55966' is disconnected

The log reports every few minutes problems with the connection but the status of the display counter is ok. The problem is rare and sporadic, log is constant.

Regards

[bouda1]

Hi,

During a such situation, did you try to keep telnet running long enough ? Is it not interrupted after several minutes ?

Another question, on your poller output you can configure the retry interval to reduce the time between too connections. Did you try to change it with a small value, 5 seconds or so ?

Regards.

[itsul]

Hi,

the last time we changed the value to 60. Since this moment, there are only 2 log entries since today night. Much better than before. Now we changed the value to 5 as you suggested.

Let`s wait a couple of days, what happen in the logs.

Regards

[itsul]

Unfortunately the log will be filled every few seconds: [Fri Oct 20 12:48:20 2017] error: feeder: error occured while processing client 'central-broker-master-input-4694': TCP peer '10.1.1.1:53796' is disconnected [Fri Oct 20 12:53:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4695': TCP peer '10.1.1.1:54894' is disconnected [Fri Oct 20 12:58:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4696': TCP peer '10.1.1.1:55998' is disconnected [Fri Oct 20 13:03:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4697': TCP peer '10.1.1.1:57098' is disconnected [Fri Oct 20 13:08:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4698': TCP peer '10.1.1.1:58194' is disconnected [Fri Oct 20 13:13:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4699': TCP peer '10.1.1.1:59290' is disconnected [Fri Oct 20 13:18:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4700': TCP peer '10.1.1.1:60386' is disconnected [Fri Oct 20 13:23:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4701': TCP peer '10.1.1.1:33250' is disconnected [Fri Oct 20 13:28:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4702': TCP peer '10.1.1.1:34356' is disconnected [Fri Oct 20 13:33:20 2017] error: feeder: error occured while processing client 'central-broker-master-input-4703': TCP peer '10.1.1.1:35458' is disconnected [Fri Oct 20 13:38:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4704': TCP peer '10.1.1.1:36564' is disconnected [Fri Oct 20 13:43:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4705': TCP peer '10.1.1.1:37662' is disconnected [Fri Oct 20 13:48:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4706': TCP peer '10.1.1.1:38758' is disconnected [Fri Oct 20 13:53:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4707': TCP peer '10.1.1.1:39852' is disconnected [Fri Oct 20 13:58:22 2017] error: feeder: error occured while processing client 'central-broker-master-input-4708': TCP peer '10.1.1.1:40950' is disconnected [Fri Oct 20 14:03:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4709': TCP peer '10.1.1.1:42050' is disconnected [Fri Oct 20 14:08:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4710': TCP peer '10.1.1.1:43150' is disconnected [Fri Oct 20 14:13:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4711': TCP peer '10.1.1.1:44244' is disconnected [Fri Oct 20 14:18:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4712': TCP peer '10.1.1.1:45344' is disconnected [Fri Oct 20 14:23:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4713': TCP peer '10.1.1.1:46438' is disconnected [Fri Oct 20 14:28:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4714': TCP peer '10.1.1.1:47534' is disconnected [Fri Oct 20 14:33:22 2017] error: feeder: error occured while processing client 'central-broker-master-input-4715': TCP peer '10.1.1.1:48630' is disconnected

[alvaro-octal]

It seems to be similar, my poller shows as not running and none of its host appear at the monitoring tab:

OS: CentOS 7.4 (AArch64) centreon-engine: 1.8.1 centreon-broker: 3.0.12

Every time I restart the poller's cbd it shows this at central cbd: /var/log/centreon-broker/central-broker-master.log (note that poller has id 2 not 1)

[1514117240] info:    storage: committing transaction
[1514117240] info:    SQL: committing transaction
[1514117240] info:    SQL: processing poller status event (id: 1, last alive: 1514117240)
[1514117240] info:    SQL: committing transaction
[1514117240] info:    storage: committing transaction
[1514117241] info:    storage: committing transaction
[1514117241] info:    SQL: committing transaction
[1514117241] info:    SQL: processing poller status event (id: 1, last alive: 1514117241)
[1514117241] info:    SQL: committing transaction
[1514117241] info:    storage: committing transaction
[1514117241] error:   feeder: error occured while processing client 'central-broker-master-input-15': TCP peer '1.2.3.4:37692' is disconnected
[1514117241] info:    feeder: thread of client 'central-broker-master-input-15' will exit
[1514117242] info:    storage: committing transaction
[1514117242] info:    SQL: committing transaction
[1514117243] info:    storage: committing transaction
[1514117243] info:    SQL: committing transaction
[1514117244] info:    storage: committing transaction
[1514117244] info:    SQL: committing transaction
[1514117244] info:    TCP: new client connected
[1514117244] info:    file: opening new file '/var/lib/centreon-broker//central-broker-master.queue.central-broker-master-input-16'
[1514117244] info:    file: end of file '/var/lib/centreon-broker//central-broker-master.queue.central-broker-master-input-16' reached, erasing file
[1514117244] info:    multiplexing: 'central-broker-master-input-16' start with 0 in queue and the queue file is disable
[1514117244] info:    feeder: thread of client 'central-broker-master-input-16' is starting
[1514117244] info:    BBDO: peer is using protocol version 2.0.0, we're using version 2.0.0
[1514117244] info:    BBDO: we have extensions 'compression' and peer has ''
[1514117244] info:    SQL: committing transaction
[1514117244] info:    storage: committing transaction

[lpinsivy]

Hi,

For your Centreon Broker output on distant poller:

• Do you enable TLS encryption? (Auto / No / Yes)
• If yes, do you use certificate?
• Do you enable negociation? (No/ yes)
• Do you enable "One peer retention"? (No / Yes)
• Do you use compression (zlib)? (Auto / No / Yes)
• Do you enable compression level and which level? (No / Yes)
• Do you enable compression buffer size and which buffer size? (No / Yes)
• Do you use IP address or FQDN to connect to Centreon central server?
• Do you define failover output?
• If yes what type of output?
• Do you define buffering timeout?

Regards,

[itsul]

Hi,

find the requested information below:

Regards

[bouda1]

The error tag is too much in those logs. The connection is lost, broker or cbd see that, the connection is established again and data are sent. Maybe we could just write a medium log information and an error could arise when the new connection fails.