loki
loki copied to clipboard
grafana
Could not run with docker-compose
Describe the bug A clear and concise description of what the bug is.
I could not run compose in production/docker
To Reproduce Steps to reproduce the behavior:
- clone this repo
cd production/dockerdocker compose up -d- container could not run healthy.
Expected behavior A clear and concise description of what you expected to happen.
Environment:
- Infrastructure: docker
- Deployment tool: docker
Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.
Please describe what error(s) you are seeing, along with what version of docker, and what operating system you are running on.
Same issue, alloy started and then exit, error log:
ts=2024-05-29T06:44:56.887806714Z level=warn msg="could not transfer logs" component_path=/ component_id=loki.source.docker.flog_scrape target=docker/39b515f95b20876a207d1a6b773340322a7be345d5f1d726ded6e6765473b459 written=0 container=39b515f95b20876a207d1a6b773340322a7be345d5f1d726ded6e6765473b459 err="context canceled"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x39e1a70]
goroutine 425 [running]:
github.com/grafana/alloy/internal/component/loki/source/docker/internal/dockertarget.(*Target).Stop(0x4002478480)
/src/alloy/internal/component/loki/source/docker/internal/dockertarget/target.go:238 +0x20
github.com/grafana/alloy/internal/component/loki/source/docker.(*tailer).Run(0x40024d5460, {0x8717cd0, 0x40024ae640})
/src/alloy/internal/component/loki/source/docker/runner.go:116 +0x260
github.com/grafana/alloy/internal/runner.(*Runner[...]).ApplyTasks.func2()
/src/alloy/internal/runner/runner.go:166 +0x98
created by github.com/grafana/alloy/internal/runner.(*Runner[...]).ApplyTasks in goroutine 188
/src/alloy/internal/runner/runner.go:163 +0x490
env:
Chip: M1
ProductName: macOS
ProductVersion: 14.5
BuildVersion: 23F79
Docker version 25.0.5, build 5dc9bcc
@rea1shane I have opened https://github.com/grafana/loki/issues/13065 to track that issue, as it appears you are running a different docker compose than the one this report is on. If I am mistaken, please let me know.
I've tried Debian and Fedora. There behaviour is same.
without change anything, i tried the compose file in production , production/docker, even in examples/getting-started. only production/docker-compose could be started.
Please describe what error(s) you are seeing, along with what version of docker, and what operating system you are running on.
I checked the logs in loki-read, first error is level=error ts=2024-06-01T07:02:19.1157467Z caller=loki.go:517 msg="module failed" module=query-scheduler-ring error="failed to start query-scheduler-ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running".
the logs in loki-write, and loki-backend shows
level=error ts=2024-06-01T07:07:25.792414429Z caller=loki.go:517 msg="module failed" module=distributor error="failed to start distributor, because it depends on module analytics, which has failed: context canceled"
level=error ts=2024-06-01T07:07:25.792426167Z caller=loki.go:517 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true 7946 false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false { false <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792435237Z caller=loki.go:517 msg="module failed" module=store error="failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true 7946 false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false { false <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792439299Z caller=loki.go:517 msg="module failed" module=analytics error="failed to start analytics, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true 7946 false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false { false <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792446405Z caller=loki.go:517 msg="module failed" module=ingester error="failed to start ingester, because it depends on module store, which has failed: context canceled"
level=info ts=2024-06-01T07:07:25.792566169Z caller=modules.go:1484 msg="server stopped"
level=info ts=2024-06-01T07:07:25.792573837Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2024-06-01T07:07:25.792582401Z caller=loki.go:506 msg="Loki stopped"
level=error ts=2024-06-01T07:07:25.792615783Z caller=log.go:230 msg="error running loki" err="failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:559\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
alert-manager logs
evel=error ts=2024-06-01T07:00:20.322Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
I'm sorry to hear you are running into an issue with this. I just tried a fresh install of the docker-compose setup in the production/docker directory on Fedora 34, and did not have any problems.
I suspect there's an error earlier on in the logs that may clue us in as to the issue. Can you capture the full logs for all of the docker containers and attach them here?
Thank you. What does your networking look like? What IPs are you using, what are your interface names?
I wonder if some of the information in this issue would help?
ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 74:56:3c:a7:d2:ad brd ff:ff:ff:ff:ff:ff inet 172.16.164.43/24 brd 172.16.164.255 scope global noprefixroute enp2s0 valid_lft forever preferred_lft forever inet6 fe80::7656:3cff:fea7:d2ad/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: br-f7837756fd18: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:e8:6e:5c:44 brd ff:ff:ff:ff:ff:ff inet 172.0.7.1/24 brd 172.0.7.255 scope global br-f7837756fd18 valid_lft forever preferred_lft forever 4: br-1728647c68c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d1:a3:18:ad brd ff:ff:ff:ff:ff:ff inet 172.0.14.1/24 brd 172.0.14.255 scope global br-1728647c68c0 valid_lft forever preferred_lft forever 5: br-2a59656a95fe: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b1:e6:74:ee brd ff:ff:ff:ff:ff:ff inet 172.0.12.1/24 brd 172.0.12.255 scope global br-2a59656a95fe valid_lft forever preferred_lft forever inet6 fe80::42:b1ff:fee6:74ee/64 scope link valid_lft forever preferred_lft forever 6: br-82336f0d9578: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d8:ce:e3:b4 brd ff:ff:ff:ff:ff:ff inet 172.0.2.1/24 brd 172.0.2.255 scope global br-82336f0d9578 valid_lft forever preferred_lft forever 8: br-ecd9d8f89caa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:8b:19:9a:41 brd ff:ff:ff:ff:ff:ff inet 172.0.3.1/24 brd 172.0.3.255 scope global br-ecd9d8f89caa valid_lft forever preferred_lft forever 9: br-17165532b731: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:25:fa:94:b4 brd ff:ff:ff:ff:ff:ff inet 172.0.73.1/24 brd 172.0.73.255 scope global br-17165532b731 valid_lft forever preferred_lft forever 10: br-5a851ebd9345: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:39:42:3f:45 brd ff:ff:ff:ff:ff:ff inet 172.0.13.1/24 brd 172.0.13.255 scope global br-5a851ebd9345 valid_lft forever preferred_lft forever 11: br-b71f53b36f98: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:39:76:30:f0 brd ff:ff:ff:ff:ff:ff inet 172.0.42.1/24 brd 172.0.42.255 scope global br-b71f53b36f98 valid_lft forever preferred_lft forever 12: br-7ad0b0e0af48: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d9:61:fd:bd brd ff:ff:ff:ff:ff:ff inet 172.0.10.1/24 brd 172.0.10.255 scope global br-7ad0b0e0af48 valid_lft forever preferred_lft forever 14: br-a30fe3aa1ff9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:90:2a:a7:c7 brd ff:ff:ff:ff:ff:ff inet 172.0.6.1/24 brd 172.0.6.255 scope global br-a30fe3aa1ff9 valid_lft forever preferred_lft forever 15: br-ef91845e7628: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:c9:c8:2d:ed brd ff:ff:ff:ff:ff:ff inet 172.0.11.1/24 brd 172.0.11.255 scope global br-ef91845e7628 valid_lft forever preferred_lft forever 16: br-cbcd2a857f95: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:64:ba:30:d4 brd ff:ff:ff:ff:ff:ff inet 172.0.4.1/24 brd 172.0.4.255 scope global br-cbcd2a857f95 valid_lft forever preferred_lft forever 17: br-df0182b5f1c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:16:b0:dd:ac brd ff:ff:ff:ff:ff:ff inet 172.0.72.1/24 brd 172.0.72.255 scope global br-df0182b5f1c0 valid_lft forever preferred_lft forever 18: br-e58ca235adba: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:33:48:d1:6e brd ff:ff:ff:ff:ff:ff inet 172.0.33.1/24 brd 172.0.33.255 scope global br-e58ca235adba valid_lft forever preferred_lft forever inet6 fe80::42:33ff:fe48:d16e/64 scope link valid_lft forever preferred_lft forever 19: br-014cef3412f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:87:b0:55:12 brd ff:ff:ff:ff:ff:ff inet 172.0.9.1/24 brd 172.0.9.255 scope global br-014cef3412f4 valid_lft forever preferred_lft forever 20: br-2bd59f3dd8de: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:01:24:f5:7e brd ff:ff:ff:ff:ff:ff inet 172.0.8.1/24 brd 172.0.8.255 scope global br-2bd59f3dd8de valid_lft forever preferred_lft forever 21: br-589f84ac3592: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:3f:2f:4c:e9 brd ff:ff:ff:ff:ff:ff inet 172.0.34.1/24 brd 172.0.34.255 scope global br-589f84ac3592 valid_lft forever preferred_lft forever inet6 fe80::42:3fff:fe2f:4ce9/64 scope link valid_lft forever preferred_lft forever 22: br-9735ec5642ea: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:38:20:4b:4c brd ff:ff:ff:ff:ff:ff inet 172.0.1.1/24 brd 172.0.1.255 scope global br-9735ec5642ea valid_lft forever preferred_lft forever 23: br-a5fdcd2ed483: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:8b:30:50:94 brd ff:ff:ff:ff:ff:ff inet 172.0.5.1/24 brd 172.0.5.255 scope global br-a5fdcd2ed483 valid_lft forever preferred_lft forever 25: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:60:de:61:03 brd ff:ff:ff:ff:ff:ff inet 172.0.0.1/24 brd 172.0.0.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:60ff:fede:6103/64 scope link valid_lft forever preferred_lft forever 2181: veth43a11ca@if2180: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether 3e:cd:ea:c9:80:b3 brd ff:ff:ff:ff:ff:ff link-netnsid 4 inet6 fe80::3ccd:eaff:fec9:80b3/64 scope link valid_lft forever preferred_lft forever 1427: vethe1b611f@if1426: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether 86:28:3e:fe:1c:7b brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::8428:3eff:fefe:1c7b/64 scope link valid_lft forever preferred_lft forever 1429: veth2099e62@if1428: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether ee:6f:8f:95:38:af brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::ec6f:8fff:fe95:38af/64 scope link valid_lft forever preferred_lft forever 1431: veth0028feb@if1430: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether 8e:36:a4:1c:91:29 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::8c36:a4ff:fe1c:9129/64 scope link valid_lft forever preferred_lft forever 1433: vethc3e9a58@if1432: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether 1a:47:3f:eb:02:f6 brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::1847:3fff:feeb:2f6/64 scope link valid_lft forever preferred_lft forever 1437: vethd85fe36@if1436: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default link/ether 22:ff:6b:7a:c0:87 brd ff:ff:ff:ff:ff:ff link-netnsid 5 inet6 fe80::20ff:6bff:fe7a:c087/64 scope link valid_lft forever preferred_lft forever 2004: br-262fbcbe84f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:1a:86:4b:88 brd ff:ff:ff:ff:ff:ff inet 172.0.21.1/24 brd 172.0.21.255 scope global br-262fbcbe84f2 valid_lft forever preferred_lft forever inet6 fe80::42:1aff:fe86:4b88/64 scope link valid_lft forever preferred_lft forever 2005: br-0798f19c15a2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:44:09:b8:b9 brd ff:ff:ff:ff:ff:ff inet 172.0.22.1/24 brd 172.0.22.255 scope global br-0798f19c15a2 valid_lft forever preferred_lft forever inet6 fe80::42:44ff:fe09:b8b9/64 scope link valid_lft forever preferred_lft forever
By utilizing the production/docker/docker-compose.yaml file, and modifying the production/docker/config/loki.yaml file to point to an invalid interface on my system, I was able to recreate what you are seeing.
I modified the loki.yaml config, and put in interface names for all of the components that mapped to what is available in docker on my system, and Loki was able to start. The contents of this are below (only changes are related to the eth0 entries)
auth_enabled: true
server:
http_listen_address: 0.0.0.0
grpc_listen_address: 0.0.0.0
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
common:
path_prefix: /loki
compactor_address: http://loki-backend:3100
replication_factor: 3
instance_interface_names:
- eth0
ring:
instance_interface_names:
- eth0
storage_config:
aws:
endpoint: minio:9000
insecure: true
bucketnames: loki-data
access_key_id: loki
secret_access_key: supersecret
s3forcepathstyle: true
memberlist:
join_members: ["loki-read", "loki-write", "loki-backend"]
dead_node_reclaim_time: 30s
gossip_to_dead_nodes_time: 15s
left_ingesters_timeout: 30s
bind_addr: ['0.0.0.0']
bind_port: 7946
gossip_interval: 2s
ingester:
lifecycler:
join_after: 10s
observe_period: 5s
interface_names:
- eth0
ring:
replication_factor: 3
kvstore:
store: memberlist
final_sleep: 0s
chunk_idle_period: 1m
wal:
enabled: true
dir: /loki/wal
max_chunk_age: 1m
chunk_retain_period: 30s
chunk_encoding: snappy
chunk_target_size: 1.572864e+06
chunk_block_size: 262144
flush_op_timeout: 10s
ruler:
enable_api: true
enable_sharding: true
wal:
dir: /loki/ruler-wal
evaluation:
mode: remote
query_frontend:
address: dns:///loki-read:9095
storage:
type: local
local:
directory: /loki/rules
ring:
instance_interface_names:
- eth0
rule_path: /loki/prom-rules
remote_write:
enabled: true
clients:
local:
url: http://prometheus:9090/api/v1/write
queue_config:
# send immediately as soon as a sample is generated
capacity: 1
batch_send_deadline: 0s
schema_config:
configs:
- from: 2020-08-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
- from: 2023-07-11
store: tsdb
object_store: s3
schema: v12
index:
prefix: index_
period: 24h
- from: 2024-01-10
store: tsdb
object_store: s3
schema: v12
index:
prefix: index_
period: 24h
- from: 2024-03-29
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
limits_config:
max_cache_freshness_per_query: '10m'
reject_old_samples: true
reject_old_samples_max_age: 30m
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
# parallelize queries in 15min intervals
split_queries_by_interval: 15m
volume_enabled: true
table_manager:
retention_deletes_enabled: true
retention_period: 336h
query_range:
# make queries more cache-able by aligning them with their step intervals
align_queries_with_step: true
max_retries: 5
parallelise_shardable_queries: true
cache_results: true
frontend:
log_queries_longer_than: 5s
compress_responses: true
max_outstanding_per_tenant: 2048
instance_interface_names:
- eth0
query_scheduler:
max_outstanding_requests_per_tenant: 1024
scheduler_ring:
instance_interface_names:
- eth0
querier:
query_ingesters_within: 2h
compactor:
working_directory: /tmp/compactor
compactor_ring:
instance_interface_names:
- eth0
There must something else blocking me. after replace config file with yours,and replace eth0 with enp2s0, I've got this:
loki-read-1 | level=error ts=2024-06-06T07:28:31.989245249Z caller=log.go:230 msg="error running loki" err="no address found for [enp2s0]\nfailed to get frontend address\ngithub.com/grafana/loki/pkg/lokifrontend/frontend.InitFrontend\n\t/src/loki/pkg/lokifrontend/frontend/config.go:52\ngithub.com/grafana/loki/pkg/loki.(*Loki).initQueryFrontend\n\t/src/loki/pkg/loki/modules.go:797\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nerror initialising module: query-frontend\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
this is my loki.yaml
auth_enabled: true
server:
http_listen_address: 0.0.0.0
grpc_listen_address: 0.0.0.0
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
common:
path_prefix: /loki
compactor_address: http://loki-backend:3100
replication_factor: 3
instance_interface_names:
- enp2s0
ring:
instance_interface_names:
- enp2s0
storage_config:
aws:
endpoint: minio:9000
insecure: true
bucketnames: loki-data
access_key_id: loki
secret_access_key: supersecret
s3forcepathstyle: true
memberlist:
join_members: ["loki-read", "loki-write", "loki-backend"]
dead_node_reclaim_time: 30s
gossip_to_dead_nodes_time: 15s
left_ingesters_timeout: 30s
bind_addr: ['0.0.0.0']
bind_port: 7946
gossip_interval: 2s
ingester:
lifecycler:
join_after: 10s
observe_period: 5s
interface_names:
- enp2s0
ring:
replication_factor: 3
kvstore:
store: memberlist
final_sleep: 0s
chunk_idle_period: 1m
wal:
enabled: true
dir: /loki/wal
max_chunk_age: 1m
chunk_retain_period: 30s
chunk_encoding: snappy
chunk_target_size: 1.572864e+06
chunk_block_size: 262144
flush_op_timeout: 10s
ruler:
enable_api: true
enable_sharding: true
wal:
dir: /loki/ruler-wal
evaluation:
mode: remote
query_frontend:
address: dns:///loki-read:9095
storage:
type: local
local:
directory: /loki/rules
ring:
instance_interface_names:
- enp2s0
rule_path: /loki/prom-rules
remote_write:
enabled: true
clients:
local:
url: http://prometheus:9090/api/v1/write
queue_config:
# send immediately as soon as a sample is generated
capacity: 1
batch_send_deadline: 0s
schema_config:
configs:
- from: 2020-08-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
- from: 2023-07-11
store: tsdb
object_store: s3
schema: v12
index:
prefix: index_
period: 24h
- from: 2024-01-10
store: tsdb
object_store: s3
schema: v12
index:
prefix: index_
period: 24h
- from: 2024-03-29
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
limits_config:
max_cache_freshness_per_query: '10m'
reject_old_samples: true
reject_old_samples_max_age: 30m
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
# parallelize queries in 15min intervals
split_queries_by_interval: 15m
volume_enabled: true
table_manager:
retention_deletes_enabled: true
retention_period: 336h
query_range:
# make queries more cache-able by aligning them with their step intervals
align_queries_with_step: true
max_retries: 5
parallelise_shardable_queries: true
cache_results: true
frontend:
log_queries_longer_than: 5s
compress_responses: true
max_outstanding_per_tenant: 2048
instance_interface_names:
- enp2s0
query_scheduler:
max_outstanding_requests_per_tenant: 1024
scheduler_ring:
instance_interface_names:
- enp2s0
querier:
query_ingesters_within: 2h
compactor:
working_directory: /tmp/compactor
compactor_ring:
instance_interface_names:
- enp2s0
If you were to not make any Loki configuration changes, for the pods that stay up, can you shell into them to see what interfaces are available? This log appears to say that enp2s0 isn't valid for your Docker networking.
For instance, my local docker setup offers eth0 and lo, and neither of those are options within my normal non-Docker environment.
Ultimately, this feels like something specific with your Docker setup, and I'm hopeful that once the correct interface is found, you can move forward.
Thank you. After I checked my config, default-address-pools is the reason.
/etc/docker/daemon.json
{
"data-root":"/home/taoqf/docker/",
"default-address-pools": [{
"base": "172.0.0.1/16",
"size": 24
}]
}
If I remove default-address-pools, with default loki.yml, I finally run the compose up.
Then, how can I fix this without change my configure?
OK, my suspicion is that the configuration you have for default address pools (thank you for finding this!) is
allocating subnets from 172.0.0.1 to 172.0.255.255, and each of those subnets is 256 IP address.
The ring code (by default) is doing some magic to find private network interfaces. The official set of 172.x.x.x private addresses is 172.16.0.0 to 172.31.255.255, which does not overlap with what you have specified.
As such, my belief is that, in order to not change your docker configuration, you'll need to modify your loki.yaml config file, like I did above, to explicitly call out eth0 for each of the components. (You should also verify that that particular interface name exists in your docker containers for Loki, and if not, change it to whatever you have configured).
OK, Thank you for your patient. I will change my docker configure. btw, modify loki.yml by adding interface name won't help. I can't figure out why loki in a container have the ability to operate local host's network. anyway, I can run loki now.