Promtail cannot harvest logs longer than 65411B and stops scanning log file

[dorinand]

Promtail stops harvesting logs from log file of docker container if log message exceed 65411B Promtail autodisover new containers with docker_sd_config configuration - it works fine. I find out, some applications does not have logs in loki and from Promtail logs, I can see:

2022-07-25T10:39:44.936526341Z level=warn ts=2022-07-25T10:39:44.936331925Z caller=target.go:195 target=docker/e406e94a86c61c8d65d80341b779f7700f3cad01cc5a0cd288c34381b8ae495e msg="finished scanning logs lines with an error" err="bufio.Scanner: token too long"

After this, the log file is no longer harvested. Tested on Docker swarm.

To Reproduce Steps to reproduce the behavior:

1. loki:2.5.0
2. promtail:2.5.0
3. Run python app that prints character as log message

import os
from time import sleep

alphabet = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']
i = 0
alphabet_len = len(alphabet)

while True:
    with open('/tmp/msg_len', 'r') as f:
        MSG_LEN = int(f.read())

    with open('/tmp/sleep', 'r') as f:
        SLEEP = int(f.read())
    char = alphabet[i % alphabet_len]

    print(char * MSG_LEN)
    i += 1
    sleep(SLEEP)

4. When MSG_LEN is 65411 its ok, 65412 stops working.

Expected behavior I would expect that Promtail is able to process larger logs, at least, to be able to configure maximum size. Also, it would be better if Promtial just log information that maximum length of log has been exceed but keeps processing other logs.

Environment:

• docker swarm
• as docker swarm service from yaml configuration file

Screenshots, Promtail config, or terminal output

1. This is message of starting harvesting container (log level should not be error I think) and when long message occur:

2022-07-25T10:39:28.243719542Z level=error ts=2022-07-25T10:39:28.243569167Z caller=target_group.go:95 msg="added Docker target" containerID=e406e94a86c61c8d65d80341b779f7700f3cad01cc5a0cd288c34381b8ae495e
2022-07-25T10:39:44.936526341Z level=warn ts=2022-07-25T10:39:44.936331925Z caller=target.go:195 target=docker/e406e94a86c61c8d65d80341b779f7700f3cad01cc5a0cd288c34381b8ae495e msg="finished scanning logs lines with an error" err="bufio.Scanner: token too long"

2. Promtail configuration

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://loki1:3100/loki/api/v1/push

scrape_configs:
- job_name: docker
  docker_sd_configs:
  - host: unix:///var/run/docker.sock
    refresh_interval: 30s
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/.*\.([0-9]{1,2})\..*'
      target_label: 'replica'
    - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
      regex: '(.*)'
      target_label: 'service'
    - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
      regex: '(.*)'
      target_label: 'stack'
  pipeline_stages:
    - static_labels:
        env: ${ENV}

[schewara]

Looks like a duplicate to https://github.com/grafana/loki/issues/6257 which should already be addressed in version 2.6

After just upgrading to 2.6.1 I don't see these errors anymore (at least on start/restart, where it used to happen)

[dorinand]

@schewara Thank you, I tested it with 1MB log line and it works. 👍