Rsyslog Filled With Closed Connections

[UnPossible-Me]

Hello, First of all, thanks you for the wonderful application you've built. It's awesome seeing just how bad my situation is, now with a nice GUI! But that's for another time,

I've got it up and running just fine, but my logs are periodically filled with big red,

omfwd: remote server at 127.0.0.1:7070 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0 try https://www.rsyslog.com/e/2027 ]

As you can imagine it's pretty obstructive, & I've got not a clue what to do, any ideas?

I've tried opening port 7070 on my firewall, & turning it off to no avail. I'm running this with docker compose (bundled version) on a Raspberry Pi 3B+, & Nginx.

Is there anything I need to fix, or just turn off logging?

[acouvreur]

Might be related to how the connection is handled in the server. Might change that

[UnPossible-Me]

I see, I look forward to it.

[acouvreur]

Might help understand the root cause https://github.com/rsyslog/rsyslog/issues/3910

[UnPossible-Me]

Reading through the thread, I'll admit I don't understand most of it, but seem to have gathered the possible solutions:

1. Using keep_alive_period=5m to reduce the spam.
2. Setting the Protocol to UDP.

Since the first option isn't available, I've tried to set the Protocol to UDP. Changing:

ports:
      - 7070:7070/udp

in the docker-compose.standalone.yml & replacing "tcp" with "udp" in /etc/rsyslog.conf

This unfortunately just seems to stop communication altogether. The containers start up just fine, but nothing is sent to ssh-log-to-influx. Nothing in logs, the docker logs just contain:

 [INFO] default - TCP Server is running on port 7070.
 [ERROR] default - connect ECONNREFUSED 172.19.0.3:8086
 [INFO] default - TCP Server is running on port 7070.
 [ERROR] default - connect ECONNREFUSED 172.19.0.4:8086
 [INFO] default - TCP Server is running on port 7070.
 [ERROR] default - connect ECONNREFUSED 172.19.0.3:8086

Using netcat -uv localhost 7070 returns Connection to localhost (::1) 7070 port [udp/*] succeeded! Typing Failed password for username from 206.253.167.10 port 11111 ssh2

No response. Nothing in logs, nor does Grafana get updated. Any clue as to what's going on?

[uniuuu]

omfwd: remote server at 10.218.99.7:7070 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0-101.el9_0.1 try https://www.rsyslog.com/e/2027 ]

[2022-10-25T13:19:48.980] [ERROR] default - No data retrieved, cannot continue
[2022-10-25T13:20:31.483] [INFO] default - CLOSED: ::ffff:10.0.0.2:60894
[2022-10-25T13:20:31.483] [INFO] default - CONNECTED: ::ffff:10.0.0.2:47116
Invalid user sdfs from 10.210.210.1 port 33935

Raising debug level for log4j showed:

[2022-10-25T13:28:34.800] [INFO] default - TCP Server is running on port 7070.
[2022-10-25T13:28:57.484] [INFO] default - CONNECTED: ::ffff:10.0.0.2:53678
[2022-10-25T13:28:57.485] [DEBUG] default - Received data Invalid user sdfs from 10.210.210.1 port 20311

Invalid user sdfs from 10.210.210.1 port 20311
[2022-10-25T13:28:57.486] [DEBUG] default - Parsed sdfs 10.210.210.1 20311
[2022-10-25T13:28:57.808] [ERROR] geohash - Unsuccessful request (200): OK { status: 'fail',
  message: 'private range',
  query: '10.210.210.1' }
[2022-10-25T13:28:57.809] [ERROR] geohash - Error: Unsuccessful request (200): [object Object]
    at retrieveLocationFromAPI (/app/dist/api.js:57:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2022-10-25T13:28:57.809] [ERROR] default - No data retrieved, cannot continue

It fails due to IP in private range.

Changing if(!data || status !== 200 || data.status !== 'success') to if (!data || status !== 200) in api.js (have get latest form container, change and then mount as volume) made it work for private IP too. See screenshot below:

This workaround is very raw so better to make permanent solution.