Build into an Dockerimage

[Happyfeet01]

Hello,

My Nextcloud runs in an docker container. Now i try to build facerecognition in this Container with an Dockerfile. When i try to compile dlib i get this error.j `FROM ubuntu:latest

RUN apt-get update && apt-get upgrade && apt-get -y install git-core cmake RUN git clone https://github.com/davisking/dlib.git && cd dlib/dlib && mkdir build && cd build RUN cmake -DBUILD_SHARED_LIBS=ON .. && make && sudo make install ` But i get this error.

Sending build context to Docker daemon 2.048kB Step 1/4 : FROM ubuntu:latest ---> a2a15febcdf3 Step 2/4 : RUN apt-get update && apt-get upgrade && apt-get -y install git-core cmake ---> Using cache ---> 5e58a14b9367 Step 3/4 : RUN git clone https://github.com/davisking/dlib.git && cd dlib/dlib && mkdir build && cd build ---> Using cache ---> ff446b74921d Step 4/4 : RUN cmake -DBUILD_SHARED_LIBS=ON .. && make && sudo make install ---> Running in 12230f7ce1e5 CMake Error: The source directory "/" does not appear to contain CMakeLists.txt. Specify --help for usage, or press the help button on the CMake GUI. The command '/bin/sh -c cmake -DBUILD_SHARED_LIBS=ON .. && make && sudo make install' returned a non-zero code: 1

What is wrong?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[agboom]

Hi @Happyfeet01, I tried to do the same, but in an Alpine container. Dlib builds fine, but pdlib fails to build in my setup.

I've opened an issue in pdlib, but no reaction so far: https://github.com/goodspb/pdlib/issues/17

[matiasdelellis]

Hi both I thought I had answered this report.. Sorry.. :sweat_smile:

I have very little experience about docker... Also I can't imagine how we can use it here, unless you have the whole instance of Nextcloud right there.. :confused: So, Please, beyond the report, I would like to know how we can use docker to help this application.

Well, That said, I will try to compile it using the dockerfile shared by @agboom in the pdlib repository. ..and I will answer if I can do something. Today, I can't help you much more.. :disappointed:

[agboom]

Hi @matiasdelellis, thanks for replying and offering to help :smiley:

I'll try to explain my motivation for building pdlib in Docker. Nextcloud provides a Docker image for their server and I'd like to try this app in a server. Since it requires the pdlib binaries and the Docker image does not include it, I'm trying to build it inside the container (or in this case in a separate container, and copy the binaries to the Nextcloud container).

Perhaps I'm overcomplicating things this way. If there is a better or faster way of providing the binaries, please let me know

[Petahh]

I support this endeavor, if it would be possible for someone who has time to compile "easy install" package for the OS which official Nexcloud image for docker is using: Linux fe5d544ca6e5 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux or introduce build procedure which will work, then it can be easily included in Dockerfile and build as an image alongside with Nextcloud application, that would be perfect!

[matiasdelellis]

Hi, Again, I have no experience in docker, and I don't feel comfortable using docker to have an instance of nextcloud.. :disappointed:

Well, The best I can offer you a dockerfile to compile and test the libraries.

FROM nextcloud:apache

# Install dlib.

RUN apt-get update ; \
    apt-get install -y build-essential wget cmake libx11-dev libopenblas-dev

ARG DLIB_BRANCH=v19.18
RUN wget -c -q https://github.com/davisking/dlib/archive/$DLIB_BRANCH.tar.gz \
    && tar xf $DLIB_BRANCH.tar.gz \
    && mv dlib-* dlib \
    && cd dlib/dlib \
    && mkdir build \
    && cd build \
    && cmake -DBUILD_SHARED_LIBS=ON --config Release .. \
    && make \
    && make install


# Install PDLib.

RUN apt-get install unzip

ARG PDLIB_BRANCH=master
RUN wget -c -q https://github.com/goodspb/pdlib/archive/$PDLIB_BRANCH.zip \
    && unzip $PDLIB_BRANCH \
    && mv pdlib-* pdlib \
    && cd pdlib \
    && phpize \
    && ./configure \
    && make \
    && make install


# Enable PDlib

RUN echo "extension=pdlib.so" > /usr/local/etc/php/conf.d/pdlib.ini


# Test PDlib instalation. (DROP IT LATTER)

RUN apt-get install -y git
RUN git clone https://github.com/matiasdelellis/pdlib-min-test-suite.git \
    && cd pdlib-min-test-suite \
    && make


# Drop development deps.

RUN apt-get -y remove build-essential wget cmake unzip git \
    && apt-get -y autoremove \
    && apt-get -y autoclean

This result on:

[matias@nube dlib-debian]$ docker build .
............DROP THE BUILD.........
......................
php scripts/face_detect.php
Welcome to pdlib min test suite for Facerecognition app...

First we try to open the models... Done

Processing file: input/Big Bang Theory.jpg
Number of faces detected: 7
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Processing file: input/Big Bang Theory.png
Number of faces detected: 7
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
Face landmarks... Done
Face descriptor... Done
 ---> 9a1e18e2fe41

Then it would work ...

[matiasdelellis]

p.s: Note that I start from the docker that you passed, so you should only compile this one, and run this instead of the previous one, or just copy the libraries compiled here to the instance you want to use.

EDIT Edit last docker file to clean packages... Then remove the test when you dont need it. :wink:

[Petahh]

Hello @matiasdelellis testing it out, right now on my instance of Docker and all seems to work well. I have simply included your lines into the original Dockerfile of Nextcloud added php memory adjustment. Removed testing sequence as it's not very wise to do that during the build.

So now you have full easy peasy installation procedure with your addon on Docker platform, which is supported almost on any platform. :)

1. Create build files "Dockerfile" and "supervisord.conf" (the first line can be adjusted to your desired version which you want to use) :

FROM nextcloud:16.0.3-apache

RUN set -ex; \
    \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        ffmpeg \
        libmagickcore-6.q16-3-extra \
        procps \
        smbclient \
        supervisor \
#       libreoffice \
    ; \
    rm -rf /var/lib/apt/lists/*

RUN set -ex; \
    \
    savedAptMark="$(apt-mark showmanual)"; \
    \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        libbz2-dev \
        libc-client-dev \
        libkrb5-dev \
        libsmbclient-dev \
    ; \
    \
    docker-php-ext-configure imap --with-kerberos --with-imap-ssl; \
    docker-php-ext-install \
        bz2 \
        imap \
    ; \
    pecl install smbclient; \
    docker-php-ext-enable smbclient; \
    \
# reset apt-mark's "manual" list so that "purge --auto-remove" will remove all build dependencies
    apt-mark auto '.*' > /dev/null; \
    apt-mark manual $savedAptMark; \
    ldd "$(php -r 'echo ini_get("extension_dir");')"/*.so \
        | awk '/=>/ { print $3 }' \
        | sort -u \
        | xargs -r dpkg-query -S \
        | cut -d: -f1 \
        | sort -u \
        | xargs -rt apt-mark manual; \
    \
    apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update ; \
    apt-get install -y build-essential wget cmake libx11-dev libopenblas-dev

RUN wget -c -q https://github.com/davisking/dlib/archive/v19.18.tar.gz \
    && tar xf v19.18.tar.gz \
    && mv dlib-* dlib \
    && cd dlib/dlib \
    && mkdir build \
    && cd build \
    && cmake -DBUILD_SHARED_LIBS=ON --config Release .. \
    && make \
    && make install


RUN apt-get install unzip

ARG PDLIB_BRANCH=master

RUN wget -c -q https://github.com/goodspb/pdlib/archive/$PDLIB_BRANCH.zip \
    && unzip $PDLIB_BRANCH \
    && mv pdlib-* pdlib \
    && cd pdlib \
    && phpize \
    && ./configure \
    && make \
    && make install


RUN echo "extension=pdlib.so" > /usr/local/etc/php/conf.d/pdlib.ini

RUN echo memory_limit=1024M > /usr/local/etc/php/conf.d/memory-limit.ini

RUN mkdir -p \
    /var/log/supervisord \
    /var/run/supervisord \
;

COPY supervisord.conf /etc/supervisor/supervisord.conf

ENV NEXTCLOUD_UPDATE=1

CMD ["/usr/bin/supervisord"]

Now create supervisord.conf:

[supervisord]
nodaemon=true
logfile=/var/log/supervisord/supervisord.log
pidfile=/var/run/supervisord/supervisord.pid
childlogdir=/var/log/supervisord/
logfile_maxbytes=50MB                           ; maximum size of logfile before rotation
logfile_backups=10                              ; number of backed up logfiles
loglevel=error

[program:apache2]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=apache2-foreground

[program:cron]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=/cron.sh

2. Build your new Nextcloud image: docker build . -t z-nextcloud:full-AI

3. After build start your services, make sure your Nextcloud application has at least 3-4GB assigned otherwise face recognition will crash.

4. Start your Nextcloud service and install face recognition addon from Nextcloud AppStore.

5. Setup the face recognition with command: docker exec -it -u www-data <name-of-your-container> ./occ face:setup

6. Run the face recognition job with command: docker exec -it -u www-data <name-of-your-container> ./occ face:background_job -u <user>

Thanks a lot for constructing the command sequence! Edit: and addon as well, it's amazing! :) you should consider adding more models, like item recognition, etc. Maybe could be optional according to what people need. Edit-2: added step of app installation which I forgot

[matiasdelellis]

Wow, @Petahh :clap: :smile: :1st_place_medal:

I understand that these files are part of your custom configuration, or these are taken from the Nextcloud Image? I ask to write a small tutorial, with the least amount of change possible about the original Nextcloud image. I insist that I have no experience, and that is why I cannot suggest much or write something officially. :sweat_smile:

Edit: and addon as well, it's amazing! :) you should consider adding more models, like item recognition, etc. Maybe could be optional according to what people need.

At some point we discussed it, and concluded that it was best to focus efforts on a single tool to gain experience, and then evaluate whether to expand it. Without promises, but soon we can discuss again.. :wink:

[Happyfeet01]

Nice, really nice :-) I Would Test it in my Testserver

[Petahh]

You have two choices how to install Nextcloud in Docker. Use their final & download ready image, (which is missing some of the features), or build your own image with Dockerfile, leveraging Dockerfile examples they provide. Second option is choice for many and that is the only way how to introduce your addon in the closest to the official procedure of Nextcloud for Docker installation. I only took their Dockerfile and added in your parts of the code. Only thing which is custom is version at the top, rest is all Nextcloud original apart from your code of course. Building image is just one part of Nextcloud Docker installation procedure. Further steps we don’t elaborate. The procedure which I have written is absolute minimum to make a working image with your addon. Hope it helps to understand it better :)

[matiasdelellis]

Great.. Thanks.. :smiley: I am reading a little and I think that for tutorial we should only point to these official guides.

• https://github.com/nextcloud/docker#adding-features

...along with this clean dockerfile that I shared (Just because it's simpler)

Maybe we can add an example in the official repo

• https://github.com/nextcloud/docker/tree/master/.examples

... and I just realized that what you originally shared was an example ... :sweat_smile: Sorry.. :wink:

[Petahh]

What you have shared previously worked nice as demonstration for the test of libraries installation, but I believe would not work at all in the real deployment scenario. That’s the whole reason I have reworked it a little :)

[matiasdelellis]

What you have shared previously worked nice as demonstration for the test of libraries installation, but I believe would not work at all in the real deployment scenario. That’s the whole reason I have reworked it a little :)

You're right, I forgot about memory and maybe more .. :wink: But I mean a small tutorial starting from an Dockerfile example (the same one you originally passed), and just showing the small diff to add.. This diff would be 'similar' to my dockerfiile. :sweat_smile:

I don't want to share a big dockerfile, where if something fails you don't know where to run .. :sweat: If we want to share a complete file, it is best to do it in that Nextcloud repository, and document it well. If you want to do something like that, more than grateful .. :smile:

[Petahh]

I get your point but I don't see a way around this really. Unfortunately, that is the procedure. I don't think people will have many issues building it now, rather you will need to keep it updated with their example cause they might change it in their repo, they tend to update it with new versions.

Edit: and I doubt they will allow us to introduce this into their example file.

[skjnldsv]

Edit: and I doubt they will allow us to introduce this into their example file.

you mean nextcloud?

[agboom]

IMHO the best way to get Nextcloud up and running with Docker is to use Docker Compose. There is a very elaborate manual in the Nextcloud Docker repo.

We could for example add a page to the Wiki for Docker installation, that refers to this chapter: https://github.com/nextcloud/docker/#running-this-image-with-docker-compose

You can run Nextcloud with Docker in one of three ways:

• With Apache, using Debian as base image
• With PHP FPM behind a proxy (e.g. Nginx), using Debian as base image
• With PHP FPM behind a proxy (e.g. Nginx), using Alpine as base image

I would recommend using Alpine as base image, since Alpine containers have a much smaller footprint and soon the installation of dlib and php7-pdlib will be much easier via the Alpine package manager.

In all three cases, you'll need a docker-compose.yml file, based on the example of either the Apache section or the FPM section.

Next edit the docker-compose.yml file, replacing:

# ...
  app:
    image: nextcloud # or nextcloud:fpm or nextcloud:fpm-alpine
# ...

with the following:

# ...
  app:
    build: ./nextcloud
# ...

This will allow you to build upon the Nextcloud base image you chose.

If you're running Nextcloud with Apache based on the Debian image, to get the FaceRecognition app working you would need to build the dlib and pdlib libraries, since they're not available via apt.

Add a Dockerfile to the ./nextcloud directory with the following contents:

FROM debian AS builder

# TODO build dlib and pdlib here

FROM nextcloud 

RUN apt install # <-- TODO add runtime dependencies

COPY --from=builder /usr/local /usr/local
COPY --from=builder /usr/lib/php7/modules/pdlib.so /usr/local/lib/php/extensions/no-debug-non-zts-20180731/

RUN echo "extension=pdlib.so" > /usr/local/etc/php/conf.d/pdlib.ini

This Dockerfile uses multi-stage builds, copying only the built binaries. This way the Nextcloud image is not polluted with sources and make dependencies. You will need Docker 17.05 or higher to use multi-stage builds.

If you're running Nextcloud with Nginx and PHP FPM, fpm-alpine is recommended.

Once https://github.com/alpinelinux/aports/pull/12174 has been merged, you should be able to run the FaceRecognition app with just the following Dockerfile in the ./nextcloud directory in your Docker Compose root:

FROM nextcloud:fpm-alpine

RUN apk --update add --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing php7-pdlib

The FaceRecognition app can be installed from the app store thanks to the wonderful work done in #1

We might want to add some documentation about how to install a cron job to regularly scan for new faces?

[matiasdelellis]

Hello everyone, Momentarily we are going to use a PDlib fork for our application.

• https://github.com/matiasdelellis/pdlib

This shouldn't affect them as much, but one of the things I hope to do is improve the documentation. For example, here is a first documentation about how to install PDlib on docker.

• https://github.com/matiasdelellis/pdlib/wiki/Docker

Well, Insist in that I have no more experience with docker, :sweat_smile: therefore I want your opinion. :smile:

Thanks, Matias

[ohthehugemanatee]

In case it helps anyone, I built https://hub.docker.com/repository/docker/ohthehugemanatee/nextcloud-dlib . It's nextcloud based on alpine (the LinuxServer version of the container), with the matiasdelellis pdlib included. No guarantees etc etc etc... except that this is the container I use. :)

Enjoy!

Source at https://github.com/ohthehugemanatee/nextcloud-dlib .

[spupuz]

In case it helps anyone, I built https://hub.docker.com/repository/docker/ohthehugemanatee/nextcloud-dlib . It's nextcloud based on alpine (the LinuxServer version of the container), with the matiasdelellis pdlib included. No guarantees etc etc etc... except that this is the container I use. :)

Enjoy!

Source at https://github.com/ohthehugemanatee/nextcloud-dlib .

Hello sorry to bother you, i have a working new nextcloud install, i'm newbie with nextcloud and docker, how can i integrate face recognition? can you explain in a simpler way or direct me to a procedure? thanks in advance i'm using the image nextcloud:latest

[matiasdelellis]

Hi @spupuz You must build your own nextcloud docker...

See. https://github.com/matiasdelellis/facerecognition/wiki/Docker

And slowly read the comments from the dockerfile.. :wink:

[spupuz]

there is not any maintainer image with these libs already incorporated?

Il giorno mer 25 nov 2020 alle ore 15:31 matiasdelellis < [email protected]> ha scritto:

Hi @spupuz https://github.com/spupuz You must build your own nextcloud docker...

See. https://github.com/matiasdelellis/facerecognition/wiki/Docker

And slowly read the comments from the dockerfile.. 😉

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matiasdelellis/facerecognition/issues/160#issuecomment-733741489, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANELQBSSU4DLHP4FCMXUODSRUIKXANCNFSM4IMOWWRA .

[matiasdelellis]

Officially nothing!. In this thread someone shared one with alpine.

[spupuz]

Officially nothing!. In this thread someone shared one with alpine.

what happen when a new docker image is released and i update it with watchtower?

[spupuz]

can't face recognition deployed as a docker mods?

https://github.com/linuxserver/docker-mods

something like that?

-e DOCKER_MODS=nc-facerecognition/nc-facerecognition:latest \

this is an example

[iamklaus]

Hey guys, I've build a docker image and published it on docker hub. I will maintain this image. Everything is build on top of the official nextcloud apache image. Feel free to use it: https://hub.docker.com/r/iamklaus/nextcloud

[Kami]

Here is an example of Dockerfile based on nextcloud:fpm-alpine image which I use for cron service in my docker compose setup which is working correctly.

FROM nextcloud:fpm-alpine

# Add php pdlib which is used for face recognition app
RUN apk --update add --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing php7-pdlib

RUN cp /usr/lib/php7/modules/pdlib.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902/
RUN cp /etc/php7/conf.d/pdlib.ini /usr/local/etc/php/conf.d/pdlib.ini

It doesn't require compilation since it uses existing upstream package.

Keep in mind that it won't work for php and php-fpm binary in fpm-alpine image if you just install the package without copying over the extension module + config (since it will be installed in default php path in /etc and not /usr/local/etc which is used by the fpm-alpine image).

[DevilsDesigns]

Hey guys, I've build a docker image and published it on docker hub. I will maintain this image. Everything is build on top of the official nextcloud apache image. Feel free to use it: https://hub.docker.com/r/iamklaus/nextcloud

i saw this im trying to figure out how to run the docker compose for this could you please specify what you used for the docker-compose.yml

[jjbeto]

Hey guys, I've build a docker image and published it on docker hub. I will maintain this image. Everything is build on top of the official nextcloud apache image. Feel free to use it: https://hub.docker.com/r/iamklaus/nextcloud

Im wondering how hard it would be to also build it for armhf arch (for raspberry pi users)..

[tclavier]

Same as @jjbeto i have build a docker image and publish it on gitlab registry : https://gitlab.com/tclavier/docker-nextcloud-facerecognition/ Everything is build on top of the official nextcloud image. Just pull registry.gitlab.com/tclavier/docker-nextcloud-facerecognition they are 4 tags available :

• :22
• :22-apache
• :apache
• :latest

I use this image for my personal nextcloud install.

Is it possible to publish the docker image directly from this github repository ? without docker hub ...

[KptnKMan]

Hey guys, I've build a docker image and published it on docker hub. I will maintain this image. Everything is build on top of the official nextcloud apache image. Feel free to use it: https://hub.docker.com/r/iamklaus/nextcloud

Hi, I'm using this docker container to try and setup face recognition. However, I get to the stage of setting up the models, and I get this error:

# sudo -u www-data php occ app:enable facerecognition

Fatal error: Allowed memory size of 2097152 bytes exhausted (tried to allocate 438272 bytes) in /var/www/html/3rdparty/composer/autoload_real.php on line 37

The container has 8GB memory assigned, and is using less than 300MB. Can anyone help?

Edit: I fixed it, there is a known issue with the official docker that is discussed in this thread on the official nextcloud project.

The commands I used to get this working (This is for my 8GB docker):

# update apt and install dependencies
apt update
apt install sudo nano
# set available memory value (best used by setting a file mapping):
echo "PHP_MEMORY_LIMIT=4096M" > /usr/local/etc/php/php.ini
# check current memory value:
sudo -u www-data php -i | grep memory_limit
# setup face recognition memory value
sudo -u www-data php occ face:setup --memory 4GB
# setup face recognition and models
sudo -u www-data php occ app:enable facerecognition
sudo -u www-data php occ face:setup -m 1
sudo -u www-data php occ face:setup -m 3
sudo -u www-data php occ face:setup -m 4

Hope this helps someone.

Also, I'm running this on Unraid 6.9.2, I can provide a working unraid template if there's interest.