prometheus_varnish_exporter
prometheus_varnish_exporter copied to clipboard
jonnenauha
Exporter running in a separate Docker container
Hi,
If I understood correctly, Docker notes mentions that you can't run the exporter in a separate container because you can't access the varnishstat.
How about to use docker exec and access the varnishstat binary from another container?
A little example on how it could be done.
docker run -it \
-v /var/run/docker.sock:/var/run/docker.sock \
ubuntu:latest \
sh -c "apt-get update ; apt-get install docker.io -y ; docker exec -it my-varnish-container varnishstat -h"
In this manner you can access the varnishtat from another container and have the metrics exposed for Prometheus scraping.
Cheers!
A more elaborate and self-explanatory version;
docker run -it \
-v /var/run/docker.sock:/var/run/docker.sock \
-e "VARNISH_CONTAINER=my-varnish.dev" \
ubuntu:latest \
sh -c " \
apt-get update \
&& apt-get install docker.io curl -y \
&& echo '#!/usr/bin/env bash' > /usr/local/bin/varnishstat \
&& echo 'docker exec -it \${VARNISH_CONTAINER} varnishstat \$@' >> /usr/local/bin/varnishstat \
&& chmod +x /usr/local/bin/varnishstat \
&& varnishstat -j \
&& curl -L -O https://github.com/jonnenauha/prometheus_varnish_exporter/releases/download/1.3.4/prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \
&& tar zxfv prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \
&& cd prometheus_varnish_exporter-1.3.4.linux-amd64 \
&& ./prometheus_varnish_exporter -test \
"
varnishstat -j is working smoothly but I'm receiving this error from the exporter:
2017/11/07 20:00:04 prometheus_varnish_exporter v1.3.4 (8aee39a) {
"ListenAddress": ":9131",
"Path": "/metrics",
"HealthPath": "",
"VarnishstatExe": "varnishstat",
"Params": {
"Instance": "",
"VSM": ""
},
"Verbose": false,
"NoExit": false,
"Test": true,
"Raw": false
}
2017/11/07 20:00:04 [FATAL] Varnish version initialize failed: exit status 1
I'm using Ubuntu just for testing, it could be shipped with a much smaller FROM Docker image.
Sorry forgot to respond to this! I still don't see this being very user friendly. The use case of running varnish and its varnish exporter in a container is probably quite rare. I understand that many prom exporters are provided as docker packages, but there is still no simple way to setup the comms between my process and varnish running in different containers.
So I'm not going to focus on this. If someone does run varnish in a container, they'll probably be more easy off by running this exporter in the same container (there is probably supported ways of running multiple processes in a single container, I don't know what exactly though) and exposing its port to the outside world for scrapes.
If you do get some progress here, you can add more info here if someone else is interested.
Hi @jonnenauha,
Actually I think most Prometheus exporters use the approach to run as a separate container, also this is Docker's best practice, to have a highly specialized container running one process and its threads. It is possible to run multiple services inside one container but I would rather to avoid it.
I have several projects running on top of Kubernetes and Docker Compose and I break up my infrastructure in layers, layers like monitoring, logging, app, database, caching, and so on... Thus, when a developer is spinning their environment they just spin-up the layers that they need to work on, there is no need to have monitoring and logging most of the time. By doing it I'm saving time and reducing cost.
What I basically showed previously is that you can access varnishstat binary running in a container from other container and I think that is basically what you need to collect Varnish metrics, isn't it ?
I did another code to show how easy it would be for users;
1 - Create a temporary folder and place inside the files below:
2 - Create a Dockerfile (this would be your Docker image) with:
FROM ubuntu:latest
RUN apt-get update \
&& apt-get install docker.io curl -y
RUN echo '#!/usr/bin/env bash' > /usr/local/bin/varnishstat \
&& echo 'docker exec -it \${VARNISH_CONTAINER} varnishstat \$@' >> /usr/local/bin/varnishstat \
&& chmod +x /usr/local/bin/varnishstat
RUN mkdir -p /opt/prometheus_varnish_exporter \
&& cd /opt/prometheus_varnish_exporter \
&& curl -L -O https://github.com/jonnenauha/prometheus_varnish_exporter/releases/download/1.3.4/prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \
&& tar zxfv prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \
&& ln -s /opt/prometheus_varnish_exporter/prometheus_varnish_exporter-1.3.4.linux-amd64/prometheus_varnish_exporter /usr/local/bin/
ENTRYPOINT ["prometheus_varnish_exporter"]
CMD ["-h"]
3 - Create a docker-compose.yaml file
version: '3'
services:
varnish:
image: million12/varnish:latest
container_name: varnish.dev
tty: true
stdin_open: true
varnish-exporter:
build: ./
container_name: varnish-exporter.dev
tty: true
stdin_open: true
depends_on:
- varnish
environment:
VARNISH_CONTAINER: varnish.dev
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command:
- -web.listen-address=9131
4 - Run the command docker-compose up
5 - See the results
In my example we are building the Docker image locally, for a user they would just need to point to your future Docker Image and set the environment variable for the Varnish service container, simple as that! :)
Cheers,
Ivan Pinatti
I think Prometheus is also used very often with Kubernetes and using exec isn't a working solution there.
I tried to figure out how varnishstat actually gets its data. Also I noticed the command line flag -N with some blurry description. Since I am not very familiar with C I did a strace varnishstat -1 and saw that varnishstat is openening the file /var/lib/varnish/$hostname/_.vsm. I tested it with varnishstat -1 -N /var/lib/varnish/$hostname/_.vsm and it worked. Also providing gibberish for -N fails with an error.
So splitting varnishd and varnishstat into two containers might be achieved by sharing that file. I will investigate further into that.
It tried to implement this, but I failed because varnishstat demanded a running varnishd. Fortunately this idea is already impemented in lswith/varnish_exporter.
I don't know why the fork is diverging from this repo.
Yeh that repo might work but its lacking a lot of features and support for new varnish versions. You can however clone it and then just paste all .go files to it from my repo, it might work if he did not mod the code heavily :)
It turns out, that my previous try did not succeed, because I had varnish v4 for the actual proxy and varnish v5 for the exporter. This means sharing /var/lib/varnish between the proxy and the exporter container just works.
Here is an example:
(the exporter and the server shouldn't be in the same image, but all the public Docker images still use Varnish v4)
FROM golang:1.10-alpine as exporter
RUN apk add --no-cache git
RUN go get github.com/jonnenauha/prometheus_varnish_exporter
WORKDIR /go/src/github.com/jonnenauha/prometheus_varnish_exporter
RUN git checkout 1.4
RUN go build
FROM alpine:3.7
RUN apk --no-cache add varnish
COPY config/default.vcl /etc/varnish/default.vcl
COPY --from=exporter /go/bin/prometheus_varnish_exporter /usr/local/bin
ENTRYPOINT /usr/sbin/varnishd
Kubernetes deployment:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: proxy
labels:
team: platform
app: proxy
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app: proxy
template:
metadata:
name: proxy
labels:
app: proxy
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9131"
spec:
containers:
- name: proxy
image: "varnish-exporter"
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
command:
- /usr/sbin/varnishd
args:
- -f/etc/varnish/default.vcl
- -F # forground
- -a0.0.0.0:80 # listen address
- -smalloc,300M # storage backend
resources:
requests:
cpu: 2
memory: 1100Mi
limits:
cpu: 2
memory: 1400Mi
volumeMounts:
- name: var
mountPath: /var/lib/varnish
- name: exporter
image: "varnish-exporter"
imagePullPolicy: Always
command:
- /usr/local/bin/prometheus_varnish_exporter
ports:
- name: metrics
containerPort: 9131
volumeMounts:
- name: var
mountPath: /var/lib/varnish
volumes:
- name: var
emptyDir: {}
Hi,
we just moved our exporter into a separate container:
version: '2'
services:
varnish:
image: xyz/sys/docker/varnish:latest
restart: always
ipc: host
network_mode: host
mem_limit: 240g
volumes:
- ./src:/etc/varnish
- varnish_tmpfs:/usr/var/varnish
environment:
BIND_PORT: :80
CACHE_SIZE: 180G
VCL_CONFIG: /etc/varnish/default.vcl
VARNISHD_PARAMS: -p default_ttl=120 -p default_grace=3600 -S /etc/varnish/secret -T 127.0.0.1:6082 -p vcl_dir=/etc/varnish/
varnishexporter:
image: xyz/sys/docker/varnish:latest
restart: always
mem_limit: 100m
ipc: host
network_mode: host
depends_on:
- varnish
volumes:
- ./src:/etc/varnish:ro
- varnish_tmpfs:/usr/var/varnish/:ro
command: prometheus_varnish_exporter
Initially it worked fine for about 1-2 days. Then Varnish reached it's container limit and got OOM-killed and restarted. From that time we only get the basic varnish-metrics:
# HELP varnish_mgt_child_died Child process died (signal)
# TYPE varnish_mgt_child_died gauge
varnish_mgt_child_died 1
# HELP varnish_mgt_child_dump Child process core dumped
# TYPE varnish_mgt_child_dump gauge
varnish_mgt_child_dump 0
# HELP varnish_mgt_child_exit Child process normal exit
# TYPE varnish_mgt_child_exit gauge
varnish_mgt_child_exit 0
# HELP varnish_mgt_child_panic Child process panic
# TYPE varnish_mgt_child_panic gauge
varnish_mgt_child_panic 0
# HELP varnish_mgt_child_start Child process started
# TYPE varnish_mgt_child_start gauge
varnish_mgt_child_start 2
# HELP varnish_mgt_child_stop Child process unexpected exit
# TYPE varnish_mgt_child_stop gauge
varnish_mgt_child_stop 0
# HELP varnish_mgt_uptime Management process uptime
# TYPE varnish_mgt_uptime gauge
varnish_mgt_uptime 164803
# HELP varnish_up Was the last scrape of varnish successful.
# TYPE varnish_up gauge
varnish_up 1
# HELP varnish_version Varnish version information
# TYPE varnish_version gauge
varnish_version{major="6",minor="0",patch="1",revision="8d54bec5330c29304979ebf2c425ae14ab80493c",version="6.0.1"} 1
After restarting the varnish-container manually it works again. Not sure where exactly the problem lies...
ignore my previous comment.. i forgot to export "VSM_NOPID" (https://varnish-cache.org/docs/trunk/reference/vsm.html#vsm-and-containers)
A more elaborate and self-explanatory version;
docker run -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -e "VARNISH_CONTAINER=my-varnish.dev" \ ubuntu:latest \ sh -c " \ apt-get update \ && apt-get install docker.io curl -y \ && echo '#!/usr/bin/env bash' > /usr/local/bin/varnishstat \ && echo 'docker exec -it \${VARNISH_CONTAINER} varnishstat \$@' >> /usr/local/bin/varnishstat \ && chmod +x /usr/local/bin/varnishstat \ && varnishstat -j \ && curl -L -O https://github.com/jonnenauha/prometheus_varnish_exporter/releases/download/1.3.4/prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \ && tar zxfv prometheus_varnish_exporter-1.3.4.linux-amd64.tar.gz \ && cd prometheus_varnish_exporter-1.3.4.linux-amd64 \ && ./prometheus_varnish_exporter -test \ "varnishstat -j is working smoothly but I'm receiving this error from the exporter:
2017/11/07 20:00:04 prometheus_varnish_exporter v1.3.4 (8aee39a) { "ListenAddress": ":9131", "Path": "/metrics", "HealthPath": "", "VarnishstatExe": "varnishstat", "Params": { "Instance": "", "VSM": "" }, "Verbose": false, "NoExit": false, "Test": true, "Raw": false } 2017/11/07 20:00:04 [FATAL] Varnish version initialize failed: exit status 1I'm using Ubuntu just for testing, it could be shipped with a much smaller FROM Docker image.
How did you corrected the "Varnish version initialize failed: exit status 1" error?
Checkout the readme, that is the now supported way of doing this. Exporter is running in host but varnish can run in a docker container.
https://github.com/jonnenauha/prometheus_varnish_exporter#docker
If you want to run also the exporter in separate container, I'm not sure how to do this. Seems you are copying varnishstat binary over to the exporter container. This does not cut it, as the binary will user unix sockets or some other mechanism to talk to the actual varnish process (not entirely sure what the route here is).
If you echo your test varnishstat -j you should see it also fails (seems to have success exit code)? Or does it get the JSON stats? If it does, then the exporter should as well. Try -verbose if you get more info about the error.
Hi,
we have the exporter running in a separate container, and its been working for about 1 year now. Just build your varnish image, and just add the exporters binary.
Then run it like this:
version: '2'
services:
varnish:
image: cktest.net/sys/docker/varnish:latest
restart: always
network_mode: host
mem_limit: 240g
volumes:
- ./src:/etc/varnish
- varnish_tmpfs:/var/lib/varnish
environment:
MALLOC_CONF: lg_dirty_mult:8,lg_chunk:18
BIND_PORT: :80
CACHE_SIZE: 180G
VCL_CONFIG: /etc/varnish/default.vcl
VARNISHD_PARAMS: -p default_ttl=120 -p default_grace=3600 -p nuke_limit=99999 -S /etc/varnish/secret -T 127.0.0.1:6082 -p vcl_dir=/etc/varnish/
varnishexporter:
image: cktest.net/sys/docker/varnish:latest
restart: always
mem_limit: 100m
network_mode: host
environment:
VSM_NOPID: 1
depends_on:
- varnish
volumes:
- ./src:/etc/varnish:ro
- varnish_tmpfs:/var/lib/varnish/:ro
command: prometheus_varnish_exporter
volumes:
varnish_tmpfs:
driver_opts:
type: tmpfs
device: tmpfs
The exporter might restart a couple of times, because varnish is not up fast enough, but after a couple of seconds you should be able to get those metrics from port 9131.
regards, Roman
Cool, thanks.
It looks like you are sharing /var/lib/varnish to both service containers. That location has the .vms file that afaik varnishstat uses to connect to varnish.
Do you have any insight how the connection part works, the /socket/varnish.socket looks to not be shared so the exporter image could connect there? What am I missing here, is there some secret sauce somewhere? :smile:
You can add prometheus_varnish_exporter -no-exit to not exit on scrape errors, Should fix that restart cycle at the beginning.
edit: the .vms file seems to have a IP:PORT combo and points to a secret file. Do the services share full networking to each other?
Also does VSM_NOPID: 1 tell the exporter container to not actually run varnish itself? Looks like its the same image, and you probably just want the varnishstat binary (and possibly others) from there?
yes we've put the .vms to a shared-location, this way we could also plug in another varnishlog-container, since we don't want to run a command like varnishlog manually inside the varnish-container.
The /socket/varnish.socket doesn't play a role in this (i removed it from the above).
That was from a test, connecting our nginx-ssl-terminator via socket instead of tcp.
ONLY the vsm is needed, and the pid-namespace info has to be ignored. (Won't work with multiple varnish-instances on the same host, but that would be a bad idea anyway i think).
See here for reference:
- https://github.com/varnishcache/varnish-cache/commit/23c08c16be38108b57bc42f3e8c8947f01bedef0
- https://varnish-cache.org/docs/5.2/reference/vsm.html
The -T -parameter is for the cli-address.
We build the exporter into the main-image for simplicity since youneed the cli-commands for both etc..
I've put up a cleaned up version here: https://github.com/strowi/varnish
Hi,
we have the exporter running in a separate container, and its been working for about 1 year now. Just build your varnish image, and just add the exporters binary.
Then run it like this:
version: '2' services: varnish: image: cktest.net/sys/docker/varnish:latest restart: always network_mode: host mem_limit: 240g volumes: - ./src:/etc/varnish - varnish_tmpfs:/var/lib/varnish environment: MALLOC_CONF: lg_dirty_mult:8,lg_chunk:18 BIND_PORT: :80 CACHE_SIZE: 180G VCL_CONFIG: /etc/varnish/default.vcl VARNISHD_PARAMS: -p default_ttl=120 -p default_grace=3600 -p nuke_limit=99999 -S /etc/varnish/secret -T 127.0.0.1:6082 -p vcl_dir=/etc/varnish/ varnishexporter: image: cktest.net/sys/docker/varnish:latest restart: always mem_limit: 100m network_mode: host environment: VSM_NOPID: 1 depends_on: - varnish volumes: - ./src:/etc/varnish:ro - varnish_tmpfs:/var/lib/varnish/:ro command: prometheus_varnish_exporter volumes: varnish_tmpfs: driver_opts: type: tmpfs device: tmpfsThe exporter might restart a couple of times, because varnish is not up fast enough, but after a couple of seconds you should be able to get those metrics from port 9131.
regards, Roman
Nice, thanks for the explanation. One more thing, how do you manage to go back to the terminal after starting the binary, without killing the process?
what terminal? There is only 1 process running per container, as it should be per docker best-practice.
We choosed to run varnish exporter as a daemon inside the varnish container, it's working fine until now.
If you're trying to run Varnish and this exporter in separate containers within the same pod in Kubernetes, you need to enable process namespace sharing and share the varnish working directory as described by @strowi above
Don't know what you mean, there is a solution in here that works if used correctly.
Not sure why this is still open...