when docker warn: "error sending batch, will retry", container can not stop

[CxistHans]

Describe the bug when docker warn: "error sending batch, will retry", container can not stop

To Reproduce Steps to reproduce the behavior:

1. Started Loki (2.9.8)
2. dockers plugin loki driver 2.9.8
3. stop loki instance
4. try to stop container: docker stop

Expected behavior when loki instance stop or error, other use loki driver service can stop and start normal.

Environment:

• Infrastructure: [docker: 20.10.8 ]
• Deployment tool: [e.g., helm, jsonnet]

Screenshots, Promtail config, or terminal output

docker log:

Jun 27 15:45:26 data-transfer-test dockerd[2764940]: time="2024-06-27T15:45:26+08:00" level=info msg="level=warn ts=2024-06-27T07:45:26.617187418Z caller=client.go:419 container_id=df24cb99aaf27d0b3e57952048cfe9bf5810cf8fcfda1228936fa6ed1b2e8344 component=client host=192.168.69.125:10030 msg=\"error sending batch, will retry\" status=-1 tenant= error=\"Post \\\"http://192.168.69.125:10030/loki/api/v1/push\\\": dial tcp 192.168.69.125:10030: connect: connection refused\"" plugin=d29798387bd4f14f18baf6744f2564e3b3b3dd365f7c032fbd9436477c800b10

service congfig:

    logging:
      driver: "loki:latest"
      options:
        mode: "non-blocking"
        max-buffer-size: "10m"
        loki-url: "http://192.168.69.125:10030/loki/api/v1/push"
        loki-timeout: 10s
        loki-retries: "3"
        keep-file: "true"
        loki-batch-size: "400"
        max-file: "100"
        max-size: "10m"
        loki-batch-wait: "10s"
        loki-min-backoff: "10s"
        loki-max-backoff: "10m"

[CxistHans]

and all unhealthy container can not restart

[CxistHans]

@cyriltovena