scenic icon indicating copy to clipboard operation
scenic copied to clipboard

Dockerfile

Open FrancescoSaverioZuppichini opened this issue 2 years ago • 7 comments

Dear all,

I hope you are doing well. Can you provide a docker file to run scenic? It looks like there are quite a lot of deps (even with tensorflow 🤢) for jax-only models. It would help the community a lot, improving usability and sharability.

Thanks,

Francesco

Hi Francesco,

A docker file sounds like a great idea! Unfortunately, I have little experience with Docker. Would you be willing to draft a docker file based on the quickstart instructions? Once we have a starting point, I'm happy to test and debug it to make it work.

Thanks, Matthias

mjlm avatar Oct 19 '22 09:10 mjlm

Ah I saw that you provided this in #533, I'll test this.

mjlm avatar Oct 19 '22 09:10 mjlm

@mjlm happy to help! Let me try to find sometime this we :)

@mjlm happy to help! Let me try to find sometime this we :)

i can provide my Dockerfile.

FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04
  
ENV TZ Asia/Seoul
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC

RUN rm /etc/apt/sources.list.d/*
RUN apt-get update && apt-get -y install \
    python3 python3-pip python3-dev \
    jupyter-notebook \
    git ssh vim

RUN apt install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get install -y python3.9

RUN ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN echo "root:root" | chpasswd
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes #prohibit-password/' /etc/ssh/sshd_config

RUN pip3 install --upgrade pip
RUN pip3 install setuptools

RUN apt-get update && apt-get install -y locales
RUN locale-gen ko_KR.UTF-8
ENV LC_ALL ko_KR.UTF-8

WORKDIR /workspace
ADD . .

RUN jupyter notebook --generate-config
RUN echo "c.NotebookApp.password = 'key'" >> /root/.jupyter/jupyter_notebook_config.py
RUN echo "c.NotebookApp.ip = '0.0.0.0'" >> /root/.jupyter/jupyter_notebook_config.py
RUN echo "c.NotebookApp.allow_root = True" >> /root/.jupyter/jupyter_notebook_config.py
RUN echo "c.NotebookApp.open_browser = False" >> /root/.jupyter/jupyter_notebook_config.py
RUN echo "c.NotebookApp.port = 8888" >> /root/.jupyter/jupyter_notebook_config.py

ENV PYTHONPATH $PYTHONPATH:/workspace

RUN chmod -R a+w /workspace
RUN alias python=python3.9

For Python 3.9 will require you to tweak it a little bit more from this script.(When I use docker, I usually pay more attention to the CUDA version.) Good luck~

BigJoon avatar Oct 27 '22 08:10 BigJoon

@BigJoon neat, thanks a lot :pray:

I have created a Dockerfile for the project with a focus on ViVit pipeline. The key challenge was to match the right versions of cudann (11), tensorrt (7). The proper cudann, tensorrt version and respective docker container version can be determined from the table available at: https://docs.nvidia.com/deeplearning/tensorrt/container-release-notes/rel-22-12.html#rel-22-12. Hopefully it helps the users of the project

FROM nvcr.io/nvidia/tensorrt:20.11-py3
# From the version table https://docs.nvidia.com/deeplearning/tensorrt/container-release-notes/rel-22-12.html#rel-22-12
# Select appropriate container version with proper cuda version and tensorrt framework version
# For me it was 20.11

RUN apt update -y
# The following libraries are need to compile and install python3.9 manually
RUN apt install build-essential zlib1g-dev libncurses5-dev \
    libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \
    libsqlite3-dev wget libbz2-dev liblzma-dev -y

# Steps to install python:3.9
WORKDIR /app
RUN wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz
RUN tar -xvf Python-3.9.7.tgz
WORKDIR /app/Python-3.9.7/
RUN ./configure --enable-optimizations
RUN make
RUN make altinstall
#RUN configure && make && make install

# Since python 3.9 is required for the project, replacing any older python3
RUN cp /usr/local/bin/python3.9 /usr/bin/python3 && \
    cp /usr/local/bin/python3.9 /usr/bin/python

# Installing pip for python3.9
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python3.9 get-pip.py

# Here we are setting up the project, with a focus on ViViT
WORKDIR /workspace
RUN cd /workspace && git clone https://github.com/google-research/scenic.git &&\
    cd scenic &&\
    sed -i '/    "tf-models-official",/d' ./setup.py &&\
    python3 -m pip install -q . &&\
    python3 -m pip install -r scenic/projects/baselines/clip/requirements.txt &&\
    python3 -m pip install scikit-image


WORKDIR /workspace/scenic
RUN pip install -r scenic/projects/vivit/requirements.txt
# manually install some libraries specifically needed by ViViT
RUN  python3 -m pip install --upgrade "jax[cuda]==0.3.1" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
RUN pip install flax==0.4.0 sklearn scikit-learn tf-models-official==2.8.0 tensorflow-gpu==2.8.0

dipodaimary avatar Jan 05 '23 14:01 dipodaimary

@dipodaimary Hi Thank you very much for your Dockerfile. I tried it. but I have warnings with the NVIDIA Driver. I have attached the warning message for your perusal. Can you help me solve this problem?

===================== == NVIDIA TensorRT ==

NVIDIA Release 20.11 (build 17147175)

NVIDIA TensorRT 7.2.1 (c) 2016-2020, NVIDIA CORPORATION. All rights reserved. Container image (c) 2020, NVIDIA CORPORATION. All rights reserved.

https://developer.nvidia.com/tensorrt

To install Python sample dependencies, run /opt/tensorrt/python/python_setup.sh

To install the open-source samples corresponding to this TensorRT release version run /opt/tensorrt/install_opensource.sh. To build the open source parsers, plugins, and samples for current top-of-tree on master or a different branch, run /opt/tensorrt/install_opensource.sh -b See https://github.com/NVIDIA/TensorRT for more information.

WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available. Use 'nvidia-docker run' to start this container; see https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .

dmrangak avatar Nov 03 '23 12:11 dmrangak