fluentd-docker-image
fluentd-docker-image copied to clipboard
fluent
Error reading logs in k8s
- fluentd or td-agent version. 1.2.0
- Environment information, e.g. OS.
K8S version 1.10.0
- Your configuration
trying to deploy using a deamonset in pre baked docker image: my docker file :
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb
FROM debian:stretch-slim
LABEL maintainer "TAGOMORI Satoshi <[email protected]>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.1"
ENV DUMB_INIT_VERSION=1.2.0
ENV GOSU_VERSION=1.10
ARG DEBIAN_FRONTEND=noninteractive
# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
ca-certificates \
ruby \
&& buildDeps=" \
make gcc g++ libc-dev \
ruby-dev \
wget bzip2 gnupg dirmngr \
" \
&& apt-get install -y --no-install-recommends $buildDeps \
&& update-ca-certificates \
&& echo 'gem: --no-document' >> /etc/gemrc \
&& gem install oj -v 3.3.10 \
&& gem install json -v 2.1.0 \
&& gem install fluentd -v 1.2.0 \
&& gem install fluent-plugin-redis \
&& dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
&& wget -O /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_$dpkgArch \
&& chmod +x /usr/bin/dumb-init \
&& wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
&& wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
&& gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
&& rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& gosu nobody true \
&& wget -O /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
&& cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
&& ./configure && make \
&& mv lib/libjemalloc.so.2 /usr/lib \
&& apt-get purge -y --auto-remove \
"Dockerfile" 77L, 2901C# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb
FROM debian:stretch-slim
LABEL maintainer "TAGOMORI Satoshi <[email protected]>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.1"
ENV DUMB_INIT_VERSION=1.2.0
ENV GOSU_VERSION=1.10
ARG DEBIAN_FRONTEND=noninteractive
# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
ca-certificates \
ruby \
&& buildDeps=" \
make gcc g++ libc-dev \
ruby-dev \
wget bzip2 gnupg dirmngr \
" \
&& apt-get install -y --no-install-recommends $buildDeps \
&& update-ca-certificates \
&& echo 'gem: --no-document' >> /etc/gemrc \
&& gem install oj -v 3.3.10 \
&& gem install json -v 2.1.0 \
&& gem install fluentd -v 1.2.0 \
&& gem install fluent-plugin-redis \
&& dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
&& wget -O /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_$dpkgArch \
&& chmod +x /usr/bin/dumb-init \
&& wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
&& wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
&& gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
&& rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& gosu nobody true \
&& wget -O /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
&& cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
&& ./configure && make \
&& mv lib/libjemalloc.so.2 /usr/lib \
&& apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# for log storage (maybe shared with host)
RUN mkdir -p /fluentd/log
# configuration/plugins path (default: copied from .)
RUN mkdir -p /fluentd/etc /fluentd/plugins
COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/
RUN chmod +x /bin/entrypoint.sh
ONBUILD COPY fluent.conf /fluentd/etc/
ONBUILD COPY plugins /fluentd/plugins/
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"
ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
ENV DUMB_INIT_SETSID 0
EXPOSE 24224 5140
ENTRYPOINT ["/bin/entrypoint.sh"]
CMD exec fluentd -c /fluentd/etc/${FLUENTD_CONF} -p /fluentd/plugins $FLUENTD_OPT
my entrypoint.sh:
#!/usr/bin/dumb-init /bin/sh
uid=${FLUENT_UID:-1000}
# check if a old fluent user exists and delete it
cat /etc/passwd | grep fluent
if [ $? -eq 0 ]; then
deluser fluent
fi
# (re)add the fluent user with $FLUENT_UID
useradd -u ${uid} -o -c "" -m fluent
export HOME=/home/fluent
# chown home and data folder
chown -R fluent /home/fluent
chown -R fluent /fluentd
exec gosu fluent "$@"
my k8s Deamonset yaml:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-logging
namespace: monitoring
labels:
name: fluentd-logging
version: v1.2.1
spec:
selector:
matchLabels:
name: fluentd-logging
template:
metadata:
labels:
name: fluentd-logging
version: v1.2.1
spec:
containers:
- name: fluentd
image: 085551151339.dkr.ecr.eu-west-1.amazonaws.com/fluentd:app
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluentd-config
mountPath: /fluentd/etc/
securityContext:
fsGroup: 1000
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluentd-config
configMap:
name: fluentd-config
serviceAccountName: fluentd-logging
my config file (fluent.conf) :
kind: ConfigMap
apiVersion: v1
metadata:
name: fluentd-config
namespace: monitoring
labels:
k8s-app: fluentd-config
data:
fluent.conf: |
@include application.conf
<match **>
@type redis
host logstash-redis.gtforge.com
port 6379
insert_key_prefix fluentd_prod
# database number is optional.
# db_number 0 # 0 is default
# If requirepass is set, please specify this.
# password hogefuga
# Users can set '${tag}' or ${tag[0]}.${tag[1]} or ...?
# insert_key_prefix '${tag}'
# Users can set strftime format.
# "%s" will be replaced into unixtime.
# "%Y%m%d.%H%M%S" will be replaced like as 20161202.112335.
# strftime_format "%s"
# Allow insert key duplicate. It will work as update values.
# allow_duplicate_key true
# ttl 300 # If 0 or negative value is set, ttl is not set in each key.
</match>
application.conf: |
<match kubernetes.var.log.containers.**fluentd**.log>
@type null
</match>
<source>
@type tail
path /var/log/containers/*.log
tag kube.*
read_from_head true
format json
<parse>
@type none
</parse>
</source>
after deploying to kubernetes pods are coming up and recieving the following error (for example):
2018-05-03 13:47:31 +0000 [warn]: #0 /var/log/containers/fluentd-logging-46phg_monitoring_fluentd-33e87e34dae2ff6bc3e98c55df9730171e756d2d9e91355b96d169838e4c2870.log unreadable. It is excluded and would be examined next time.
I've tried to play with a lot, can't find any solution...
Please advise.
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}