blueoil
blueoil copied to clipboard
Published
20 hours ago •
blue-oil
blue-oil
make build takes > 30min
In current maste branch, First make build takes more than 30min.
master
Result of time make build at current master b9db83d
Successfully tagged blueoil_iizuka:v0.11.0-10-gb9db83d
real 32m51.528s
user 0m1.168s
sys 0m0.716s
v0.11.0
Result of v0.11.0 tag
Successfully tagged blueoil_iizuka:v0.11.0
real 2m53.142s
user 0m0.300s
sys 0m0.360s
This is reasonable?
Too long, I think. We should take some action to reduce build time.
From sample result of buildkite log, It was total: 17min, install openmpi: 9min, install horovod: 2min, these are depends on CPU performance (may slower on your environment), so we should try to reduce the time of them.
[2019-08-16T08:29:08Z] Step 1/50 : FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04
[2019-08-16T08:29:08Z] Step 2/50 : MAINTAINER [email protected]
[2019-08-16T08:29:08Z] Step 3/50 : EXPOSE 6006
[2019-08-16T08:29:08Z] Step 4/50 : ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:${LD_LIBRARY_PATH}
[2019-08-16T08:29:08Z] Step 5/50 : RUN echo "deb http://ftp.jaist.ac.jp/ubuntu/ xenial main restricted universe multiverse \ndeb-src http://ftp.jaist.ac.jp/ubuntu/ xenial main restricted universe multiverse \ndeb http://ftp.jaist.ac.jp/ubuntu/ xenial-updates main restricted universe multiverse \ndeb-src http://ftp.jaist.ac.jp/ubuntu/ xenial-updates main restricted universe multiverse \ndeb http://ftp.jaist.ac.jp/ubuntu/ xenial-backports main restricted universe multiverse \ndeb-src http://ftp.jaist.ac.jp/ubuntu/ xenial-backports main restricted universe multiverse \ndeb http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse \ndeb-src http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse" > /etc/apt/sources.list
[2019-08-16T08:29:08Z] Step 6/50 : RUN apt-get update && apt-get install -y cmake locales python3 python3-dev python3-pip python3-wheel && apt-get clean && rm -rf /var/lib/apt/lists/*
[2019-08-16T08:29:08Z] Step 7/50 : RUN apt-get update && apt-get install -y python3-pil libjpeg8-dev zlib1g-dev python3-matplotlib liblapack-dev git make build-essential libssl-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev && apt-get clean && rm -rf /var/lib/apt/lists/*
[2019-08-16T08:29:08Z] Step 8/50 : RUN apt-get update && apt-get install -y crossbuild-essential-arm64
[2019-08-16T08:29:08Z] Step 9/50 : RUN locale-gen en_US.UTF-8
[2019-08-16T08:29:08Z] Step 10/50 : ENV LANG en_US.UTF-8
[2019-08-16T08:29:08Z] Step 11/50 : ENV LANGUAGE en_US:en
[2019-08-16T08:29:08Z] Step 12/50 : ENV LC_ALL en_US.UTF-8
[2019-08-16T08:29:08Z] Step 13/50 : ENV PYENV_ROOT /usr/local/pyenv
[2019-08-16T08:29:08Z] Step 14/50 : RUN git clone https://github.com/yyuu/pyenv.git $PYENV_ROOT
[2019-08-16T08:29:08Z] Step 15/50 : RUN git clone https://github.com/yyuu/pyenv-virtualenv.git $PYENV_ROOT/plugins/pyenv-virtualenv
[2019-08-16T08:29:08Z] Step 16/50 : ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
[2019-08-16T08:29:08Z] Step 17/50 : RUN eval "$(pyenv init -)" && eval "$(pyenv virtualenv-init -)"
[2019-08-16T08:29:08Z] Step 18/50 : ARG python_version="3.6.3"
[2019-08-16T08:29:08Z] Step 19/50 : RUN pyenv install ${python_version} && pyenv virtualenv -p python${python_version%.*} ${python_version} python${python_version%.*} && pyenv global python${python_version%.*}
[2019-08-16T08:29:08Z] Step 20/50 : RUN pip install -U pip setuptools
[2019-08-16T08:29:08Z] Step 21/50 : RUN apt-get update && apt-get install -y g++-5-arm-linux-gnueabihf && ln -s /usr/bin/arm-linux-gnueabihf-g++-5 /usr/bin/arm-linux-gnueabihf-g++
[2019-08-16T08:29:08Z] Step 22/50 : COPY lmnet/*requirements.txt /tmp/requirements/
[2019-08-16T08:29:08Z] Step 23/50 : RUN pip install -r /tmp/requirements/gpu.requirements.txt
[2019-08-16T08:33:04Z] Step 24/50 : RUN pip uninstall -y prompt-toolkit
[2019-08-16T08:33:07Z] Step 25/50 : COPY lmnet/third_party /home/blueoil/lmnet/third_party
[2019-08-16T08:33:07Z] Step 26/50 : RUN cd /home/blueoil/lmnet/third_party/coco/PythonAPI && pip install -e .
[2019-08-16T08:33:18Z] Step 27/50 : RUN mkdir /tmp/openmpi && cd /tmp/openmpi && wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-4.0.1.tar.gz && tar zxf openmpi-4.0.1.tar.gz && cd openmpi-4.0.1 && ./configure --enable-orterun-prefix-by-default && make -j $(nproc) all && make install && ldconfig && rm -rf /tmp/openmpi
[2019-08-16T08:42:26Z] Step 28/50 : RUN apt-get install -y --no-install-recommends openssh-client openssh-server && mkdir -p /var/run/sshd
[2019-08-16T08:42:32Z] Step 29/50 : ENV HOROVOD_GPU_ALLREDUCE NCCL
[2019-08-16T08:42:33Z] Step 30/50 : ENV HOROVOD_WITH_TENSORFLOW 1
[2019-08-16T08:42:33Z] Step 31/50 : RUN ldconfig /usr/local/cuda-10.0/targets/x86_64-linux/lib/stubs
[2019-08-16T08:42:34Z] Step 32/50 : RUN pip install -r /tmp/requirements/dist.requirements.txt
[2019-08-16T08:44:52Z] Step 33/50 : RUN ldconfig
[2019-08-16T08:44:54Z] Step 34/50 : RUN echo "hwloc_base_binding_policy = none" >> /usr/local/etc/openmpi-mca-params.conf && echo "rmaps_base_mapping_policy = slot" >> /usr/local/etc/openmpi-mca-params.conf && echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
[2019-08-16T08:44:55Z] Step 35/50 : RUN mkdir /.horovod && chmod 777 /.horovod
[2019-08-16T08:44:56Z] Step 36/50 : COPY blueoil /home/blueoil/blueoil
[2019-08-16T08:44:57Z] Step 37/50 : COPY setup.* /home/blueoil/
[2019-08-16T08:44:57Z] Step 38/50 : COPY output_template /home/blueoil/output_template
[2019-08-16T08:44:57Z] Step 39/50 : WORKDIR /home/blueoil
[2019-08-16T08:44:58Z] Step 40/50 : RUN python setup.py install
[2019-08-16T08:45:14Z] Step 41/50 : RUN chmod 777 /home/blueoil
[2019-08-16T08:45:16Z] Step 42/50 : COPY dlk /home/blueoil/dlk
[2019-08-16T08:45:25Z] Step 43/50 : WORKDIR /home/blueoil/dlk
[2019-08-16T08:45:25Z] Step 44/50 : RUN PYTHONPATH=python/dlk python setup.py install
[2019-08-16T08:45:45Z] Step 45/50 : RUN chmod 777 /home/blueoil/dlk
[2019-08-16T08:45:47Z] Step 46/50 : COPY lmnet /home/blueoil/lmnet
[2019-08-16T08:45:49Z] Step 47/50 : ENV PYTHONPATH $PYTHONPATH:/home/blueoil:/home/blueoil/lmnet:/home/blueoil/dlk/python/dlk
[2019-08-16T08:45:49Z] Step 48/50 : WORKDIR /home/blueoil
[2019-08-16T08:45:49Z] Step 49/50 : RUN apt-get update && apt-get install -y x11-apps imagemagick
[2019-08-16T08:46:39Z] Step 50/50 : RUN apt-get clean && rm -rf /var/lib/apt/lists/*
I will work on this by https://github.com/blue-oil/blueoil/pull/374 Sorry for being late
@kchygoe How is this progress ? I thinks we should move installing openmpi to before pip install -r gpu.requirements.txt .
Is compiling openmpi really required ? It takes most part of whole time for building. If it's possible to remove it, it's better.
