cloudflared icon indicating copy to clipboard operation
cloudflared copied to clipboard

Published 20 hours ago • verified publisher icon cloudflare

🐛 Missing prometheus metrics

Open wasabipesto opened this issue 2 years ago • 2 comments
trafficstars

Describe the bug I have been using cloudflared tunnels for a while now, and am very happy with the experience. I have started bringing in some metrics to Prometheus for Grafana to view as a dashboard. I have been able to use quite a lot of the provided metrics, but a few that I believe should be active are not showing up for me at all.

In particular, I am looking for the metric cloudflared_tunnel_user_hostnames_counts. I can see that it's listed as an available metric in October 2020 and again in June 2022. I can see the metric is created here and the logic hasn't been edited since November 2020. However, the metric does not show in the output of any of my machines.

To Reproduce Machine 1:

justin@celebrimbor:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:38:05 celebrimbor systemd[1]: Starting cloudflared...
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Starting tunnel tunnelID=9cb1055b-b6b3-4331-bd47-5008bec287b9
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Version 2022.11.1
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Settings: map[metrics:0.0.0.0:8754 token:*****]
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Generated Connector ID: 63b24db6-5322-4fa6-9307-69bd8f7c142f
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Initial protocol quic
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF ICMP proxy will use 192.168.1.13 as source for IPv4
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF ICMP proxy will use 2600:1700:4559:8430::1cb in zone enp5s0 as source for IPv6
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022/12/15 11:38:06 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:06Z INF Connection 8f5e238f-fef0-4f0d-93ed-2423d788571c registered with protocol: quic connIndex=0 ip=198.41.200.13 location=ORD
Dec 15 11:38:06 celebrimbor systemd[1]: Started cloudflared.
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:06Z INF Connection 66bb17f6-93bf-4267-8244-0c4cf23d0eb1 registered with protocol: quic connIndex=1 ip=198.41.192.77 location=MSP
Dec 15 11:38:07 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:07Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"calibre.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8083\"},{\"hostname\":\"prowlarr.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9696\"},{\"hostname\":\"syncthing.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8384\"},{\"hostname\":\"hass.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8123\"},{\"hostname\":\"hassio.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8123\"},{\"hostname\":\"radarr.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:7878\"},{\"hostname\":\"code-c.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8443\"},{\"hostname\":\"sonarr.[redacted]\",\"service\":\"http://localhost:8989\"},{\"hostname\":\"rtorrent.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8085\"},{\"hostname\":\"vnc-c.[redacted]\",\"originRequest\":{},\"service\":\"rdp://localhost:5901\"},{\"hostname\":\"maxifold.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9025\"},{\"hostname\":\"kbt.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:1313\"},{\"hostname\":\"nextcloud.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4431\"},{\"hostname\":\"manifold.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"plex.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:32400\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=37
Dec 15 11:38:07 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:07Z INF Connection 9407f2dd-33f2-44ad-a595-f532f4cfc2c4 registered with protocol: quic connIndex=2 ip=198.41.200.23 location=ORD
Dec 15 11:38:08 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:08Z INF Connection 9773cdfc-5650-4ec3-a8ce-976037df6013 registered with protocol: quic connIndex=3 ip=198.41.192.27 location=MSP
justin@celebrimbor:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@celebrimbor:~$ curl -s localhost:8754/metrics | grep "host"
justin@celebrimbor:~$

Machine 2:

justin@ereinion:~$ justin@ereinion:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:49:01 ereinion systemd[1]: Starting cloudflared...
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Starting tunnel tunnelID=078f38bd-3a0c-44ab-bd01-d9c3eee26071
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Version 2022.11.1
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Settings: map[metrics:0.0.0.0:8754 no-autoupdate:true token:*****]
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Generated Connector ID: d6fb3312-7108-4f35-a410-a7e1f3a4860d
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Initial protocol quic
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF ICMP proxy will use 167.99.56.166 as source for IPv4
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF ICMP proxy will use 2604:a880:800:10::6c2:a001 in zone eth0 as source for IPv6
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022/12/15 11:49:02 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Connection 4821df16-d264-4830-a090-33d48a97c7b5 registered with protocol: quic connIndex=0 ip=198.41.200.23 location=ORD
Dec 15 11:49:02 ereinion systemd[1]: Started cloudflared.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Connection 10bf319f-3ff2-4c72-a5ff-30c6154947f4 registered with protocol: quic connIndex=1 ip=198.41.192.7 location=EWR
Dec 15 11:49:03 ereinion cloudflared[3165002]: 2022-12-15T16:49:03Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"prometheus-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9090\"},{\"hostname\":\"grafana-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"element.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8183\"},{\"hostname\":\"kirkwall.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:30000\"},{\"hostname\":\"requests.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:5055\"},{\"hostname\":\"jupyter.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8888\"},{\"hostname\":\"paperless-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8000\"},{\"hostname\":\"synapse.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8008\"},{\"hostname\":\"code-e.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8443\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=84
Dec 15 11:49:03 ereinion cloudflared[3165002]: 2022-12-15T16:49:03Z INF Connection da5abd30-597f-4af5-9e49-8d78e801c1cd registered with protocol: quic connIndex=2 ip=198.41.200.73 location=ORD
Dec 15 11:49:04 ereinion cloudflared[3165002]: 2022-12-15T16:49:04Z INF Connection 44d3b9ca-4878-4951-93e2-38e33095e12e registered with protocol: quic connIndex=3 ip=198.41.192.77 location=EWR
justin@ereinion:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@ereinion:~$ curl -s localhost:8754/metrics | grep "host"
justin@ereinion:~$

Machine 3:

justin@kalak:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:53:27 kalak systemd[1]: Starting cloudflared...
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Starting tunnel tunnelID=0749b8a6-9268-42bf-9164-715e6caa8efc
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Version 2022.11.1
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Settings: map[edge-ip-version:6 metrics:0.0.0.0:8754 token:*****]
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Generated Connector ID: 7316437a-b41a-45cb-bbca-afd4a1594ddd
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Initial protocol quic
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF ICMP proxy will use 0.0.0.0 as source for IPv4
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 0 is not between ping group 1 to 0"
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 0 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022/12/15 11:53:27 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Connection a891c90e-7c5d-4405-b516-9bc3f43f45da registered with protocol: quic connIndex=0 ip=2606:4700:a8::8 location=VIE
Dec 15 11:53:27 kalak systemd[1]: Started cloudflared.
Dec 15 11:53:28 kalak cloudflared[3725871]: 2022-12-15T16:53:28Z INF Connection 52169dff-9e1a-4271-ae1f-840a3b5752cb registered with protocol: quic connIndex=1 ip=2606:4700:a0::5 location=FRA
Dec 15 11:53:28 kalak cloudflared[3725871]: 2022-12-15T16:53:28Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"code-k.[redacted]\",\"service\":\"http://localhost:8443\"},{\"hostname\":\"kirkwall-beta.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:30000\"},{\"hostname\":\"prometheus.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9090\"},{\"hostname\":\"stream.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8084\"},{\"hostname\":\"grafana.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"tautulli.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8181\"},{\"hostname\":\"requests-beta.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:5055\"},{\"hostname\":\"rss.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8524\"},{\"hostname\":\"rss-bridge.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3060\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=35
Dec 15 11:53:29 kalak cloudflared[3725871]: 2022-12-15T16:53:29Z INF Connection f78f61cb-6fb6-4898-865b-137dac49eb0d registered with protocol: quic connIndex=2 ip=2606:4700:a8::1 location=VIE
Dec 15 11:53:30 kalak cloudflared[3725871]: 2022-12-15T16:53:30Z INF Connection 502bc161-5036-47bd-9db8-6f2f6150de7b registered with protocol: quic connIndex=3 ip=2606:4700:a0::2 location=FRA
Dec 15 11:54:50 kalak cloudflared[3725871]: 2022-12-15T16:54:50Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=77a0b415be9603fc-ORD ingressRule=8 originService=http://localhost:8443
Dec 15 11:54:50 kalak cloudflared[3725871]: 2022-12-15T16:54:50Z ERR Request failed error="Incoming request ended abruptly: context canceled" connIndex=1 dest=https://code-k.[redacted]/stable-6261075646f055b99068d3688932416f2346dd3b/vscode-remote-resource?path=%2Fapp%2Fcode-server%2Flib%2Fvscode%2Fextensions%2Ftheme-kimbie-dark%2Fthemes%2Fkimbie-dark-color-theme.json&tkn= ip=2606:4700:a0::5 type=http
Dec 15 11:54:51 kalak cloudflared[3725871]: 2022-12-15T16:54:51Z ERR Failed to handle QUIC stream error=EOF connIndex=1 ip=2606:4700:a0::5
justin@kalak:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@kalak:~$ curl -s localhost:8754/metrics | grep "host"
justin@kalak:~$

All 3 of the above were configured via the dashboard. To see if this was a dashboard-specific issue, I started a tunnel via the local configuration:

justin@kalak:~$ cloudflared tunnel --config /opt/cloudflared/kalak.yml --metrics 0.0.0.0:8755 --edge-ip-version 6 run
2022-12-15T17:01:14Z INF Starting tunnel tunnelID=6c72ed50-4f6e-498a-a8a5-283891dcfc61
2022-12-15T17:01:14Z INF Version 2022.11.1
2022-12-15T17:01:14Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
2022-12-15T17:01:14Z INF Settings: map[config:/opt/cloudflared/kalak.yml cred-file:/home/justin/.cloudflared/6c72ed50-4f6e-498a-a8a5-283891dcfc61.json credentials-file:/home/justin/.cloudflared/6c72ed50-4f6e-498a-a8a5-283891dcfc61.json edge-ip-version:6 metrics:0.0.0.0:8755]
2022-12-15T17:01:14Z INF cloudflared will not automatically update if installed by a package manager.
2022-12-15T17:01:14Z INF Generated Connector ID: a360f7f2-b2ab-454d-b631-c208d7707e19
2022-12-15T17:01:14Z INF Initial protocol quic
2022-12-15T17:01:14Z INF ICMP proxy will use 0.0.0.0 as source for IPv4
2022-12-15T17:01:14Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2022-12-15T17:01:14Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1000 is not between ping group 1 to 0"
2022-12-15T17:01:14Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2022-12-15T17:01:14Z INF Starting Hello World server at 127.0.0.1:43443
2022-12-15T17:01:14Z INF Starting metrics server on [::]:8755/metrics
2022/12/15 12:01:14 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-12-15T17:01:15Z INF Connection 110621e7-7bde-48f3-866d-7c2ab504fc7d registered with protocol: quic connIndex=0 ip=2606:4700:a8::6 location=VIE
2022-12-15T17:01:16Z INF Connection c9267e82-1c8d-4454-8eed-07967c58be71 registered with protocol: quic connIndex=1 ip=2606:4700:a0::8 location=FRA
2022-12-15T17:01:17Z INF Connection 17df35be-a741-4620-b04b-5b0fe8a92e9d registered with protocol: quic connIndex=2 ip=2606:4700:a8::8 location=VIE
2022-12-15T17:01:18Z INF Connection 39463562-97a1-433a-b587-4f30109271ae registered with protocol: quic connIndex=3 ip=2606:4700:a0::1 location=FRA
[leave running...]
justin@kalak:~$ curl -s localhost:8755/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@kalak:~$ curl -s localhost:8755/metrics | grep "host"
justin@kalak:~$

My assumption is that this metric is only enabled under specific circumstances, but I cannot tell what those would be. I may be missing some configuration option to make this metric available or some part of this setup may be incompatible with the metric I'm looking for.

Expected behavior I expect the metric cloudflared_tunnel_user_hostnames_counts (or a metric with a similar function) to be available for use with Prometheus.

Environment and versions

  • OS: Ubuntu 20.04.5 LTS, Ubuntu 20.04.5 LTS, Debian GNU/Linux 11 (bullseye)
  • Architecture: x86_64, x86_64, x86_64
  • Version:
    • cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)
    • cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)
    • cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)

Logs and errors See above

Additional context N/A

wasabipesto avatar Dec 15 '22 17:12 wasabipesto

Since a while I am not seeing quic_client_(sent|receive)_bytes{conn_index="[0-9]+"} anymore, this has worked before, just not sure which version broke it. I am currently using 2023.8.2.

When I grep for quic_ in the metrics I get only the following results, the traffic stats are missing:

# HELP quic_client_closed_connections Number of connections that has been closed
# TYPE quic_client_closed_connections counter
quic_client_closed_connections 2
# HELP quic_client_dropped_packets Number of bytes that have been dropped on a connection
# TYPE quic_client_dropped_packets counter
quic_client_dropped_packets{conn_index="0",packet_type="1_rtt",reason="duplicate"} 1956
quic_client_dropped_packets{conn_index="1",packet_type="1_rtt",reason="duplicate"} 56
quic_client_dropped_packets{conn_index="2",packet_type="1_rtt",reason="duplicate"} 503
# HELP quic_client_latest_rtt Latest RTT measured on a connection
# TYPE quic_client_latest_rtt gauge
quic_client_latest_rtt{conn_index="0"} 1
quic_client_latest_rtt{conn_index="1"} 16
quic_client_latest_rtt{conn_index="2"} 1
quic_client_latest_rtt{conn_index="3"} 12
# HELP quic_client_lost_packets Number of packets that have been lost from a connection
# TYPE quic_client_lost_packets counter
quic_client_lost_packets{conn_index="0",reason="reordering"} 20918
quic_client_lost_packets{conn_index="0",reason="timeout"} 3424
quic_client_lost_packets{conn_index="1",reason="reordering"} 4619
quic_client_lost_packets{conn_index="1",reason="timeout"} 58
quic_client_lost_packets{conn_index="2",reason="reordering"} 14866
quic_client_lost_packets{conn_index="2",reason="timeout"} 2726
quic_client_lost_packets{conn_index="3",reason="reordering"} 7327
quic_client_lost_packets{conn_index="3",reason="timeout"} 33
# HELP quic_client_min_rtt Lowest RTT measured on a connection in millisec
# TYPE quic_client_min_rtt gauge
quic_client_min_rtt{conn_index="0"} 0
quic_client_min_rtt{conn_index="1"} 16
quic_client_min_rtt{conn_index="2"} 0
quic_client_min_rtt{conn_index="3"} 12
# HELP quic_client_packet_too_big_dropped Count of packets received from origin that are too big to send to the edge and are dropped as a result
# TYPE quic_client_packet_too_big_dropped counter
quic_client_packet_too_big_dropped 0
# HELP quic_client_smoothed_rtt Calculated smoothed RTT measured on a connection in millisec
# TYPE quic_client_smoothed_rtt gauge
quic_client_smoothed_rtt{conn_index="0"} 0
quic_client_smoothed_rtt{conn_index="1"} 16
quic_client_smoothed_rtt{conn_index="2"} 0
quic_client_smoothed_rtt{conn_index="3"} 12
# HELP quic_client_total_connections Number of connections initiated. For all quic metrics, client means the side initiating the connection
# TYPE quic_client_total_connections counter
quic_client_total_connections 6

incognico avatar Oct 30 '23 11:10 incognico

Since a while I am not seeing quic_client_(sent|receive)_bytes{conn_index="[0-9]+"} anymore, this has worked before, just not sure which version broke it. I am currently using 2023.8.2.

When I grep for quic_ in the metrics I get only the following results, the traffic stats are missing:

It appears to have broken with the 2023.5.1 release - I didn't see your note before I filed this bug:

https://github.com/cloudflare/cloudflared/issues/1098

darron avatar Nov 03 '23 19:11 darron