Loki container alway restart

[nguyentan1980]

Hi, I start homer7-docker with server hom7-prom-all and I have problem with Loki container. It alway restart. Error log : level=error ts=2019-06-05T06:26:51.286856691Z caller=main.go:46 msg="error loading config" filename=/etc/loki/loki-local-config.yaml err="parsing time "0" as "2006-01-02": cannot parse "0" as "2006"

[lmangani]

I just updated the default Loki config, could you pull the latest changes and try again?

[nguyentan1980]

Hello Imangani, I already pulled the lastest changes and now Loki work fine. But at the error log, we still have an error at log:

promtail | level=error ts=2019-06-05T08:47:35.888553001Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log

[lmangani]

Is this a one off error or recurring? I don' t see any settings for that in Loki

[nguyentan1980]

Hi Imangani, Thanks for quick response. These message show every 30 s . promtail | level=error ts=2019-06-05T09:04:15.888588718Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:04:25.888611951Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:04:35.888557084Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:04:45.888559058Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:04:55.888568321Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:05:05.888542175Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log heplify-server | 2019/06/05 09:05:11.942938 server.go:283: INFO stats since last 5 minutes. PPS: 23, HEP: 6971, Filtered: 0, Error: 0 promtail | level=error ts=2019-06-05T09:05:15.888527983Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:05:25.888569187Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:05:35.88860587Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log promtail | level=error ts=2019-06-05T09:05:45.888562601Z caller=filetarget.go:272 msg="failed to tail file" error="file is a directory" filename=/var/log

[lmangani]

I wonder if this is a scraper trying to read.... although none are configured. could you insert an empty scrape_configs: block in your configuration?

[nguyentan1980]

Hi Imangani,

Can you show me detail about "insert an empty scrape_configs" ? I'm new member in this field. Thank you.

[lmangani]

Sorry - i was referring to the loki configuration mounted by the container. The log appears to be from a 'scraper' of sorts. In a couple issues on the Loki repository i found a reference to this setting so I was suggesting to add an empty block with this in your yml and see if we can override this (supposed) default?

[gil-obradors]

Hello,

this working for me: ( modify loki-local-config.yaml) But I can't explain why, maybe config versions

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 15m

schema_config:
  configs:
  - from: 2019-01-01
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
      prefix: index_
      period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false

[lmangani]

Thanks @gil-obradors - If other users can confirm we can set this as the default

[PNRxA]

I opened a related issue on the other repository in regards to this: https://github.com/sipcapture/heplify-server/issues/261

The from and ring fields need modifying for newer versions of Loki as stated above

edit: looks like this was fixed for -all, submitted pull request for -loki-

[denzs]

Problem still exists... It doesnt matter if i use the loki-local-config.yaml from the repo or the one from above..

[lmangani]

@denzs have you tried with no config?

[denzs]

I did:

• docker-compose down
• mv loki/loki-local-config.yaml /tmp/
• docker-compose up But nothing changed...

[lmangani]

Without fresh error logs it's hard to help.

[denzs]

I know.. but to me there is very little information in the logs...

[0][root@siptrace:~]# docker logs loki 
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
[0][root@siptrace:~]#

I already removed the whole homer7-docker directory and did a fresh checkout to make sure, there are no fragments left... But the result is the same... That is very strange, because it did work for some days before...

[quchenyuan]

I know.. but to me there is very little information in the logs...

[0][root@siptrace:~]# docker logs loki 
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
[0][root@siptrace:~]#

I already removed the whole homer7-docker directory and did a fresh checkout to make sure, there are no fragments left... But the result is the same... That is very strange, because it did work for some days before...

Hello,

this working for me: ( modify loki-local-config.yaml) chunk_store_config: max_look_back_period: 0s

table_manager: retention_deletes_enabled: false retention_period: 0s

pay attention to max_look_back_period and retention_period, values should be end with an 's'.

[JnMik]

Same here

[JnMik]

Got it working by editing the configs I had in a kubernetes secret

[shuklaneerajdev]

I know.. but to me there is very little information in the logs...

[0][root@siptrace:~]# docker logs loki 
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
failed parsing config: /etc/loki/local-config.yaml: not a valid duration string: "0"
[0][root@siptrace:~]#

I already removed the whole homer7-docker directory and did a fresh checkout to make sure, there are no fragments left... But the result is the same... That is very strange, because it did work for some days before...

Hello,

this working for me: ( modify loki-local-config.yaml) chunk_store_config: max_look_back_period: 0s

table_manager: retention_deletes_enabled: false retention_period: 0s

pay attention to max_look_back_period and retention_period, values should be end with an 's'.

This solution worked for me as well. Thanks.