problem adding plugin to the image

[rolele]

I get this stack while trying to build the modified Dockerfile from the README to install new plugin

[root@manager2 fluentd]# docker build -t aa/fluentdaggregator .
Sending build context to Docker daemon 180.2 kB
Step 1 : FROM fluent/fluentd:latest-onbuild
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
 ---> Using cache
Step 1 : COPY plugins /fluentd/plugins/
 ---> Using cache
 ---> cf7bc09254a9
Step 2 : MAINTAINER YOUR_NAME <...@...>
 ---> Running in 1e2d0e21fdff
 ---> de2f1246900c
Removing intermediate container 1e2d0e21fdff
Step 3 : WORKDIR /home/fluent
 ---> Running in 4b6a30b4ac32
 ---> a99f8f4acbb5
Removing intermediate container 4b6a30b4ac32
Step 4 : ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
 ---> Running in 6a0651a5ede5
 ---> 294b972adb14
Removing intermediate container 6a0651a5ede5
Step 5 : USER root
 ---> Running in fa18f5ce97cf
 ---> 592201f0f7dc
Removing intermediate container fa18f5ce97cf
Step 6 : RUN apk --no-cache --update add sudo build-base ruby-dev &&     sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer &&     rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && sudo -ufluent gem sources -c &&     apk del sudo build-base ruby-dev &
& rm -rf /var/cache/apk/*
 ---> Running in 50436c80cd6f
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  build-base (missing):
    required by: world[build-base]
  ruby-dev (missing):
    required by: world[ruby-dev]
  sudo (missing):
    required by: world[sudo]
The command '/bin/sh -c apk --no-cache --update add sudo build-base ruby-dev &&     sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer &&     rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem&& sudo -u fluent gem sources -c &&     apk del sudo build-base
 ruby-dev && rm -rf /var/cache/apk/*' returned anon-zero code: 3

here is the Dockerfile (exactly similar as the one in README)

FROM fluent/fluentd:latest-onbuild
MAINTAINER YOUR_NAME <...@...>
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH

# cutomize following "gem install fluent-plugin-..." line as you wish

USER root
RUN apk --no-cache --update add sudo build-base ruby-dev && \

    sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer && \

    rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && sudo -u fluent gem sources -c && \
    apk del sudo build-base ruby-dev && rm -rf /var/cache/apk/*

EXPOSE 24284

USER fluent
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

I created a plugins folder and pulled fluent-plugin-elasticsearch and fluent-plugin-record-reformer in it.

[repeatedly]

How about replace fluent gem with fluent-gem? It seems a typo of README...

[repeatedly]

Ah, wait. fluent is for -u. Please ignore above comment.

[repeatedly]

It seems the problem is alpine linux or cdn side: https://github.com/gliderlabs/docker-alpine/issues/155

[rolele]

Ok I decided to use the ubuntu image but I still have some troubles.

FROM fluent/fluentd:ubuntu-base
USER root
WORKDIR /fluentd/plugins/fluent-plugin-elasticsearch-master
RUN gem build fluent-plugin-elasticsearch.gemspec
RUN gem install fluent-plugin-elasticsearch-1.7.0.gem
RUN fluent-gem install fluent-plugin-elasticsearch
WORKDIR /fluentd/plugins/fluent-plugin-record-reformer-master
RUN gem build fluent-plugin-record-reformer.gemspec
RUN gem install fluent-plugin-record-reformer-0.8.2.gem
RUN fluent-gem install fluent-plugin-record-reformer
USER ubuntu
WORKDIR /home/ubuntu

here is the result of running fluent manually from within the container

[root@manager2 fluentd]# docker run -it -v /etc/fluentd:/fluentd/etc aa/fluentdaggregator bash
root@fe45fdebaed2:/home/ubuntu# fluent-gem install fluent-plugin-elasticsearch
Successfully installed fluent-plugin-elasticsearch-1.7.0
Parsing documentation for fluent-plugin-elasticsearch-1.7.0
Done installing documentation for fluent-plugin-elasticsearch after 0 seconds
1 gem installed
root@fe45fdebaed2:/home/ubuntu# fluentd -c /fluentd/etc/fluent_aggregator.conf -p /fluentd/plugins
2016-10-18 04:07:15 +0000 [info]: reading config file path="/fluentd/etc/fluent_aggregator.conf"
2016-10-18 04:07:15 +0000 [info]: starting fluentd-0.12.29
2016-10-18 04:07:15 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.7.0'
2016-10-18 04:07:15 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.2'
2016-10-18 04:07:15 +0000 [info]: gem 'fluentd' version '0.12.29'
2016-10-18 04:07:15 +0000 [info]: adding match pattern="*.**" type="copy"
2016-10-18 04:07:15 +0000 [error]: config error file="/fluentd/etc/fluent_aggregator.conf" error="Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins"
2016-10-18 04:07:15 +0000 [info]: process finished code=256
2016-10-18 04:07:15 +0000 [warn]: process died within 1 second. exit.
root@fe45fdebaed2:/home/ubuntu#

I think this is a permission issue because the gems are installed while being the root user. If I try to install it while being the ubuntu user I do not have the permission to install the gem.

How can I install plugin with the ubuntu image please? I found it very difficult to work with

[rolele]

here is the stack when I try to install the gem plugins as the ubuntu user

[root@manager2 fluentd]# docker build -t aa/fluentdaggregator .
Sending build context to Docker daemon 179.7 kB
Step 1 : FROM fluent/fluentd:ubuntu-base
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
 ---> Using cache
Step 1 : COPY plugins /fluentd/plugins/
 ---> Using cache
 ---> f520160b942a
Step 2 : WORKDIR /fluentd/plugins/fluent-plugin-elasticsearch-master
 ---> Using cache
 ---> 94fd8bd916a1
Step 3 : RUN gem build fluent-plugin-elasticsearch.gemspec
 ---> Running in 6b874c5f157d
fatal: Not a git repository (or any of the parent directories): .git
WARNING:  description and summary are identical
WARNING:  open-ended dependency on fluentd (>= 0.10.43) is not recommended
  if fluentd is semantically versioned, use:
    add_runtime_dependency 'fluentd', '~> 0.10', '>= 0.10.43'
WARNING:  open-ended dependency on excon (>= 0) is not recommended
  if excon is semantically versioned, use:
    add_runtime_dependency 'excon', '~> 0'
WARNING:  open-ended dependency on rake (>= 0, development) is not recommended
  if rake is semantically versioned, use:
    add_development_dependency 'rake', '~> 0'
WARNING:  pessimistic dependency on test-unit (~> 3.1.0, development) may be overly strict
  if test-unit is semantically versioned, use:
    add_development_dependency 'test-unit', '~> 3.1', '>= 3.1.0'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
ERROR:  While executing gem ... (Errno::EACCES)
    Permission denied @ rb_sysopen - fluent-plugin-elasticsearch-1.7.0.gem
The command '/bin/sh -c gem build fluent-plugin-elasticsearch.gemspec' returned a non-zero code: 1
[root@manager2 fluentd]#