loki icon indicating copy to clipboard operation
loki copied to clipboard

Log lines with same timestamp are not in order after ingesting to Loki

Open duj4 opened this issue 8 months ago • 4 comments

Describe the bug I am ingesting the log files to Loki via Fluent-bit loki output plugin, the log lines with same timestamp were not in the order as they were in the original logs.

Original log files:

[D 2024-06-19 17:25:02.417 26256 17660 - - ldap3:118] BASIC:instantiated Tls: <Tls(validate=<VerifyMode.CERT_NONE: 0>)> [D 2024-06-19 17:25:02.417 26256 17660 - - ldap3:118] BASIC:instantiated Server: <Server(host=‘shgmwcdc01.cod.ms.com.cn’, port=636, use_ssl=True, allowed_referral_hosts=[(‘’, True)], tls=Tls(validate=<VerifyMode.CERT_NONE: 0>), get_info=‘SCHEMA’, mode=‘IP_V6_PREFERRED’)> [D 2024-06-19 17:25:02.417 26256 17660 - - ldap3:118] ERROR:unknown strategy for <ldaps://shgmwcdc01.cod.ms.com.cn:636 - ssl - user: itsm_api_qa - not lazy - unbound - closed - - tls not started - not listening - No strategy - internal decoder> [D 2024-06-19 17:25:02.417 26256 17660 - - ldap3:118] BASIC:instantiated : <ldaps://shgmwcdc01.cod.ms.com.cn:636 - ssl - user: itsm_api_qa - not lazy - unbound - closed - - tls not started - not listening - No strategy - internal decoder - async - real DSA - not pooled - cannot stream output> [D 2024-06-19 17:25:02.417 26256 17660 - - ldap3:118] BASIC:instantiated Connection: <Connection(server=Server(host=‘shgmwcdc01.cod.ms.com.cn’, port=636, use_ssl=True, allowed_referral_hosts=[(‘’, True)], tls=Tls(validate=<VerifyMode.CERT_NONE: 0>), get_info=‘SCHEMA’, mode=‘IP_V6_PREFERRED’), user=‘itsm_api_qa’, auto_bind=‘NO_TLS’, version=3, authentication=‘SIMPLE’, client_strategy=‘RESTARTABLE_ASYNC’, auto_referrals=True, check_names=True, read_only=False, lazy=False, raise_exceptions=False, fast_decoder=True, auto_range=True, receive_timeout=15, return_empty_attributes=True, auto_encode=True, auto_escape=True, use_referral_cache=False)>

Queried from Grafana: image

Output with Fluent-bit stdout plugin: image

Per the suggestion in cannot order loki entries with same timestamp · Issue #56071 · grafana/grafana · GitHub, I set increment_duplicate_timestamp as true, but still no luck.

To Reproduce Steps to reproduce the behavior:

  1. Started Loki
  2. Ingested log snippet above by retrieving the timestamp via regex

Expected behavior Log lines with same timestamp should be kept in order as they were in the original log files.

Environment:

  • Infrastructure: K8S
  • Deployment tool: helm
  • Loki Version: 2.9.8

Screenshots, Promtail config, or terminal output Uploaded in above sections.

duj4 avatar Jun 24 '24 11:06 duj4