Slow Tails log to Grafana loki, and sudden pause for a long time

[junedirajagukguk]

I Installed fluentbit in standalone server, its quite normal when i started the fluentbit , but it's so long to see the current timestamp and logs time match, maybe because the file logs quite big, (more than 500mb)

like the image above, i started the fluentbit for a half hour, but only get the logs until 7am when the current time is 4:39 pm, and when i look yesterday explore logs, its only read the log like at 5am and stop till 1 hour like the image below

Here's my fluentbit.conf

SERVICE] flush 1 daemon Off log_level info parsers_file parsers.conf plugins_file plugins.conf http_server Off http_listen 0.0.0.0 http_port 2020 storage.metrics on

[INPUT] name tail tag orion-gateway Path /path/to/log Path_Key filename Read_from_head true multiline.parser multiline-regex-mule Buffer_Max_Size 500MB Rotate_Wait 10

[INPUT] name tail tag orion-gateway Path /path/to/log Path_Key filename Read_from_head true multiline.parser multiline-regex-mule Buffer_Max_Size 500MB Rotate_Wait 10

[FILTER] Name multiline Match orion-gateway* multiline.key_content log multiline.parser multiline-regex-mule

[OUTPUT] name loki match orion-gateway host
port 443 tls on tls.verify on labels job=log, instance=gateway, $filename http_user
http_passwd

Please Help me to make the logs tail normal and live. i actually have another fluentbit and its running normal, i guess maybe because the file log is under 200mb not like my current problem.

[patrick-stephens]

Please follow the issue template to capture all the relevant information - particularly things like the OS and version of FB used? Also, are your parsers correctly parsing the timestamps? Are you expecting the timestamp from the log line (which depends on your application) or the time from when Fluent Bit handles it? Remember Loki also has a restriction on ingesting logs that are too old (as indicated by their timestamp).

Personally I would take Loki out of the situation and look at the stdout output directly to see what Fluent Bit is handling and when, along with the actual timestamp it is using.

[junedirajagukguk]

Please follow the issue template to capture all the relevant information - particularly things like the OS and version of FB used? Also, are your parsers correctly parsing the timestamps? Are you expecting the timestamp from the log line (which depends on your application) or the time from when Fluent Bit handles it? Remember Loki also has a restriction on ingesting logs that are too old (as indicated by their timestamp).

Personally I would take Loki out of the situation and look at the stdout output directly to see what Fluent Bit is handling and when, along with the actual timestamp it is using.

Halo, Im using Almalinux 8.8 and fluent-bit-2.2.1 , i expect the logs that sent to loki not take that very long time. even stop. i only set a single log every input which is today log only.

ahh i see , i'll try . thx for the suggestion.

anw, here is my parsers conf.

[PARSER] Name apache Format regex Regex ^(?[^ ]) [^ ] (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S)?)?" (?[^ ]) (?[^ ])(?: "(?[^"])" "(?[^"])")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?[^ ]) [^ ] (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^ ]) +\S)?" (?[^ ]) (?[^ ])(?: "(?[^"])" "(?.)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^[[^ ]* (?[^]])] [(?[^]])](?: [pid (?[^]])])?( [client (?[^]])])? (?.)$ [PARSER] Name nginx Format regex Regex ^(?[^ ]) (?[^ ]) (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S*)?)?" (?[^ ]) (?[^ ])(?: "(?[^"])" "(?[^"])") Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name k8s-nginx-ingress Format regex Regex ^(?[^ ]) - (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S*)?)?" (?[^ ]) (?[^ ]) "(?[^"])" "(?[^"])" (?<request_length>[^ ]) (?<request_time>[^ ]) [(?<proxy_upstream_name>[^ ])] ([(?<proxy_alternative_upstream_name>[^ ])] )?(?<upstream_addr>[^ ]) (?<upstream_response_length>[^ ]) (?<upstream_response_time>[^ ]) (?<upstream_status>[^ ]) (?<reg_id>[^ ]).$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name docker-daemon Format regex Regex time="(?[^ ])" level=(?[^ ]) msg="(?[^ ].)" Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name syslog-rfc5424 Format regex Regex ^<(?[0-9]{1,5})>1 (?[^ ]+) (?[^ ]+) (?[^ ]+) (?[-0-9]+) (?[^ ]+) (?([(.?)]|-)) (?.+)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On [PARSER] Name syslog-rfc3164-local Format regex Regex ^<(?[0-9]+)>(?[^ ]* {1,2}[^ ]* [^ ]) (?[a-zA-Z0-9_/.-])(?:[(?[0-9]+)])?(?:[^:]:)? (?.)$ Time_Key time Time_Format %b %d %H:%M:%S Time_Keep On [PARSER] Name syslog-rfc3164 Format regex Regex /^<(?[0-9]+)>(?[^ ] {1,2}[^ ]* [^ ]) (?[^ ]) (?[a-zA-Z0-9_/.-])(?:[(?[0-9]+)])?(?:[^:]:)? (?.)$/ Time_Key time Time_Format %b %d %H:%M:%S Time_Keep On [PARSER] Name mongodb Format regex Regex ^(?[^ ])\s+(?\w)\s+(?[^ ]+)\s+[(?[^]]+)]\s+(?.?) (?(\d+))?(:?ms)?$ Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On Time_Key time [PARSER] Name envoy Format regex Regex ^[(?<start_time>[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S)?)? (?\S+)" (?[^ ]) (?<response_flags>[^ ]) (?<bytes_received>[^ ]) (?<bytes_sent>[^ ]) (?[^ ]) (?<x_envoy_upstream_service_time>[^ ]) "(?<x_forwarded_for>[^ ])" "(?<user_agent>[^"])" "(?<request_id>[^"])" "(?[^ ])" "(?<upstream_host>[^ ])" Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On Time_Key start_time [PARSER] Name istio-envoy-proxy Format regex Regex ^[(?<start_time>[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S)?)? (?\S+)" (?<response_code>[^ ]) (?<response_flags>[^ ]) (?<response_code_details>[^ ]) (?<connection_termination_details>[^ ]) (?<upstream_transport_failure_reason>[^ ]) (?<bytes_received>[^ ]) (?<bytes_sent>[^ ]) (?[^ ]) (?<x_envoy_upstream_service_time>[^ ]) "(?<x_forwarded_for>[^ ])" "(?<user_agent>[^"])" "(?<x_request_id>[^"])" (?[^ ])" "(?<upstream_host>[^ ])" (?<upstream_cluster>[^ ]) (?<upstream_local_address>[^ ]) (?<downstream_local_address>[^ ]) (?<downstream_remote_address>[^ ]) (?<requested_server_name>[^ ]) (?<route_name>[^ ]) Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On Time_Key start_time [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]) (?.)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On [PARSER] Name kube-custom Format regex Regex (?[^.]+)?.?(?<pod_name>a-z0-9?(?:.a-z0-9?))(?<namespace_name>[^]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64}).log$ [MULTILINE_PARSER] name multiline-regex-mule type regex flush_timeout 1000 rule "start_state" "/^(\d)./" "cont" rule "cont" "/(?:^[ ,\t,a-z,A-Z,,-])./" "cont"

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.