fluent-bit
fluent-bit copied to clipboard
fluent
Multiline is processing lines out of order
Bug Report
My setup consists of a multi-container docker-compose project that's sending logs to journald. The services are mostly Python. I've got multi-line working mostly fine, but in some cases it seems to break in a way that I can't fully understand. I've pasted an example below.
fluent-bit.conf
[INPUT]
Name systemd
Path /var/log/journal
Systemd_Filter _SYSTEMD_UNIT=docker.service
Tag untagged
[FILTER]
Name multiline
Match untagged
Multiline.key_content MESSAGE
Multiline.parser multiline-traceback-regex,python
Buffer true
[OUTPUT]
Name stdout
Match untagged
parses.conf
[MULTILINE_PARSER]
name multiline-traceback-regex
type regex
flush_timeout 5000
rule "start_state" "^\d{4}-" "cont"
rule "cont" "^(?!\d{4}-)" "cont"
Sample logs below. These are from journalctl CONTAINER_NAME=my_container but the actual logs are being ingested by the systemd input, and are JSON formatted.
2024-05-02 19:46:37,444 INFO [root] [base_daemon.py:26] - Starting up Signals daemon
2024-05-02 19:46:37,452 WARNING [root] [signals.py:64] - Couldn't get a login token from App!
2024-05-02 19:46:37,454 WARNING [root] [signals.py:66] - Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 158, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
httplib_response = self._make_request(conn, method, url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 167, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f32c653f410>: Failed to establish a new connection: [Errno -2] Name or service not known
<snip> (this goes on for quite a while)
2024-05-02 19:46:37,461 INFO [root] [signals.py:395] - Fetched 0 active output connectors
2024-05-02 19:46:37,965 INFO [root] [signals.py:395] - suppressing duplicate messages...
Snippet of output from fluent-bit
[0] untagged: [[1714679197.444539000, {}], {"MESSAGE"=>"2024-05-02 19:46:37,444 INFO [root] [base_daemon.py:26] - Starting up Signals daemon", "CONTAINER_NAME"=>"my_container"}]
[1] untagged: [[1714679197.452401000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"2024-05-02 19:46:37,452 WARNING [root] [signals.py:64] - Couldn't get a login token from App!"}]
[2] untagged: [[1714679197.455503000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
httplib_response = self._make_request(conn, method, url,"}]
[3] untagged: [[1714679197.455503000, {}], {"MESSAGE"=>" File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 167, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f32c653f410>: Failed to establish a new connection: [Errno -2] Name or service not known"}]
[4] untagged: [[1714679197.456147000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen("}]
<snip>
As you can see, the traceback starting [2] untagged: doesn't match the log output (compare the subsequent lines):
File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 158, in _new_conn
vs.
File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
What am I doing wrong here?
This is on Fluent Bit v3.0.2 (Git commit: 33ce918351cdae2056b9e1e470ee163293f3fb5d)
