cellxgene_VIP
cellxgene_VIP copied to clipboard
cellxgene_VIP deploying with Heroku
Hi, I would like to deploy the cellxgene VIP result with Heroku
and would like to know how to modify the "Dockerfile" provided by cellxgene
Here is my modified version and I'm getting this error
FROM ubuntu:bionic
ENV LC_ALL=C.UTF-8 ENV LANG=C.UTF-8
RUN apt-get update &&
apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests &&
apt-get install git -y
git clone https://github.com/interactivereport/cellxgene_VIP.git
COPY . .
RUN pip install -r requirements.txt
RUN conda env create -n VIP -f VIP.macOS.yml
RUN conda init bash
RUN conda activate VIP
RUN echo "Make sure flask is installed:" RUN python -c "import flask"
COPY run.py . RUN ./config.macOS.sh
ENTRYPOINT ["cellxgene"]
and getting below errors
E: Unable to locate package clone E: Unable to locate package https://github.com/interactivereport E: Couldn't find any package by glob 'https://github.com/interactivereport' E: Couldn't find any package by regex 'https://github.com/interactivereport' The command '/bin/sh -c apt-get update && apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests && apt-get install git -y git clone https://github.com/interactivereport/cellxgene_VIP.git' returned a non-zero code: 100 ▸ Error: docker build exited with Error: 100
Thanks,
I would love to see how this works
Dear @skim245, @FADHLyemen and cellxgene_vip developer,
I am a PhD student at CNAG-CRG evaluating right now the potential use of cellxgene/cellxgene_VIP for 3TR (https://www.3tr-imi.eu/)
Since I am also struggling with setting up a functional Dockerfile I thought it could be beneficial to share what I tried so far in your issue. Maybe we could help each other.
Best regards, Ivo
Best shot so far:
#Command Line Interface in cellxgene_VIP is not working
#rpy2 runs into an error (reproduce: import rpy2.robjects as ro)
FROM rocker/tidyverse:4.0.3
RUN apt-get update && apt-get -y install --no-install-recommends \
git \
#dependency of igraph \
libglpk40 \
#dependecy of Cairo \
libxt-dev \
#dependencies for ./config.sh \
cpio \
npm
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_4.9.2-Linux-x86_64.sh -O ~/miniconda.sh && \
bash ~/miniconda.sh -b -p /opt/conda && rm ~/miniconda.sh
ENV PATH="/opt/conda/bin:${PATH}"
RUN git clone https://github.com/interactivereport/cellxgene_VIP.git
WORKDIR /cellxgene_VIP
RUN conda update conda && \
conda config --set channel_priority flexible && \
conda env update -n base -f VIP.yml
#clone and modify cellxgene
RUN ./config.sh
#faster R package installer
RUN R -e "install.packages('pak',repos='https://r-lib.github.io/p/pak/dev')"
RUN R -e " \
install_packages_or_die <- function (pkgs,repos='http://cran.us.r-project.org/'){ \
for (e in pkgs){ \
pak::pkg_install(e); \
if ( ! library(e, character.only=TRUE, logical.return=TRUE) ) { \
stop ('ERROR: failed installing requested package \'',e,'\'') } } }; \
pkgs <- new.env(hash=T, parent=emptyenv()); \
install_packages_or_die (c('fgsea','ggpubr','ggrastr','Seurat','viridis'))"
Did not manage to install Seurat:
#Not working because package of conflicts /w Seurat
FROM continuumio/miniconda3:4.9.2
RUN apt-get update && apt-get install -y \
cpio \
git \
xfonts-base \
npm
ADD data/* /data/
RUN git clone https://github.com/interactivereport/cellxgene_VIP.git
WORKDIR /cellxgene_VIP
RUN conda update conda && \
conda config --set channel_priority flexible && \
conda env update -n base -f VIP_conda_R.yml
ENV LIBARROW_MINIMAL=false
#needs to be installed with conda because r is installed via conda
RUN conda install -c conda-forge \
r-cairo=1.5-10 \
r-ragg=0.3.1
#r-seurat #creates package conflicts with devtools
RUN ./config.sh
RUN R -e 'if(!require(devtools)) install.packages("devtools",repos = "http://cran.us.r-project.org")'
RUN R -e " \
install_packages_or_die <- function (pkgs, repos='http://cran.us.r-project.org/'){ \
for (e in ls(pkgs)){ \
devtools::install_version(e,version=pkgs[[e]],repos=repos); \
if ( ! library(e, character.only=TRUE, logical.return=TRUE) ) { \
stop ('ERROR: failed installing requested package \'',e,'\'') } } }; \
pkgs <- new.env(hash=T, parent=emptyenv()); \
assign('foreign','0.8-76', pkgs); \
assign('ggpubr','0.3.0', pkgs); \
assign('ggrastr','0.1.9', pkgs); \
assign('arrow','2.0.0', pkgs); \
assign('rmarkdown','2.5', pkgs); \
assign('tidyverse','1.3.0', pkgs); \
assign('viridis','0.5.1', pkgs); \
assign('BiocManager','1.30.10', pkgs); \
install_packages_or_die (pkgs)"
RUN R -e 'if(!require(fgsea)) BiocManager::install("fgsea")'
#RUN R -e 'if(!require(Seurat)) devtools::install_version("Seurat",version="3.2.3",repos = "http://cran.us.r-project.org")'
#package conflicts
Seurat working but this time I did not manage to install fgsea:
#Not working because of package conflicts /w fgsea
#FROM continuumio/anaconda3:2021.04
FROM continuumio/miniconda3:4.9.2
RUN apt-get update && apt-get install -y \
cpio \
git \
npm
ADD data/* /data/
RUN git clone https://github.com/interactivereport/cellxgene_VIP.git
WORKDIR /cellxgene_VIP
RUN conda update conda && \
conda config --set channel_priority flexible && \
conda env update -n base -f VIP_conda_R.yml
#overwrite r-base (3.6.3/4.0.4/4.0.5 does not work -> package conflicts)
#packages needs to be installed with conda because R is installed via conda
RUN conda install -c conda-forge \
r-base=4.0.2 \
r-cairo=1.5_12.2 \
r-ragg=0.3.1 \
r-ggrastr=0.2.3 \
r-arrow=2.0.0 \
r-biocmanager=1.30.15 \
r-foreign=0.8_80 \
r-ggpubr=0.4.0 \
r-rmarkdown=2.8 \
r-seurat=3.2.2 \
r-tidyverse=1.3.0 \
r-viridis=0.6.1
#Trying to install fgsea
#RUN conda install -c bioconda bioconductor-fgsea
#Error multiple package conflicts
#RUN R -e 'if(!require(fgsea)) BiocManager::install("fgsea")'
#Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) :
# namespace ‘ellipsis’ 0.3.1 is already loaded, but >= 0.3.2 is required
#modifies cellxgene
RUN ./config.sh
@IvoLeist thank you for sharing this. I've been trying to deploy cellxgene_VIP with Heroku using below codes
git add heroku.yml git commit -m "Add heroku.yml" git push heroku master
and getting an error code below
remote: =!= There were problems parsing your heroku.yml. We've detected the following issues: remote: remote: build.docker.web references a Dockerfile that does not exist
any thoughts or suggestion?
Thanks,
@skim245 without seeing your heroku.yml and knowing your folder structure I cannot help you with that.
But my random guess is that you have specified the wrong path inside build.docker.web If your heroku.yml is in the root directory and the Dockerfile is e.g. inside a folder named app:
build:
docker:
web: app/Dockerfile
If the heroku.yml and the Dockerfile are in the same folder:
build:
docker:
web: Dockerfile
@IvoLeist here is my heroku.yml
build: docker: web: Dockerfile
run: web: command: - cellxgene launch --host localhost --port 5050 --disable-annotations --verbose 042221_scRNA_data.h5ad and both of my heroku.yml and the Dockerfile are in the same folder. and do you know which extension to use for Dockerfile ?
I saw the discussion about Dockerfile in this issue, so I'm sharing my Dockerfile here too.
Half of the file is copied from https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile and half of the file is from @IvoLeist. I failed to use conda to solve the R dependencies directly, I got the same errors written in @IvoLeist 's Dockerfile. Instead, I followed the Jupyter Notebook Dockerfile and used Mamba to solve the dependencies. It worked pretty well.
I didn't try deploying to Heroku but I managed to deploy on Google Cloud Run. I believe they work in the same way.
FROM debian:bullseye
# ref: https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile
# Install all OS dependencies for notebook server that starts but lacks all
# features (e.g., download as all possible file formats)
# - tini is installed as a helpful container entrypoint that reaps zombie
# processes and such of the actual executable we want to start, see
# https://github.com/krallin/tini#why-tini for details.
# - apt-get upgrade is run to patch known vulnerabilities in apt-get packages as
# the ubuntu base image is rebuilt too seldom sometimes (less than once a month)
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update --yes && \
apt-get upgrade --yes && \
apt-get install --yes --no-install-recommends \
git cpio build-essential python3 python3-pip cmake libhdf5-dev cmake llvm npm nodejs jq \
tini wget ca-certificates sudo locales fonts-liberation && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
# Configure environment
ENV CONDA_DIR=/opt/conda \
SHELL=/bin/bash \
LC_ALL=en_US.UTF-8 \
LANG=en_US.UTF-8 \
LANGUAGE=en_US.UTF-8
ENV PATH="${CONDA_DIR}/bin:${PATH}"
# Install conda as jovyan and check the sha256 sum provided on the download site
WORKDIR /tmp
ARG PYTHON_VERSION="3.8.12"
# CONDA_MIRROR is a mirror prefix to speed up downloading
# For example, people from mainland China could set it as
# https://mirrors.tuna.tsinghua.edu.cn/github-release/conda-forge/miniforge/LatestRelease
ARG CONDA_MIRROR=https://github.com/conda-forge/miniforge/releases/latest/download
# ---- Miniforge installer ----
# Check https://github.com/conda-forge/miniforge/releases
# Package Manager and Python implementation to use (https://github.com/conda-forge/miniforge)
# We're using Mambaforge installer, possible options:
# - conda only: either Miniforge3 to use Python or Miniforge-pypy3 to use PyPy
# - conda + mamba: either Mambaforge to use Python or Mambaforge-pypy3 to use PyPy
# Installation: conda, mamba, pip
RUN set -x && \
# Miniforge installer
miniforge_arch=$(uname -m) && \
miniforge_installer="Mambaforge-Linux-${miniforge_arch}.sh" && \
wget --quiet "${CONDA_MIRROR}/${miniforge_installer}" && \
/bin/bash "${miniforge_installer}" -f -b -p "${CONDA_DIR}" && \
rm "${miniforge_installer}" && \
# Conda configuration see https://conda.io/projects/conda/en/latest/configuration.html
conda config --system --set auto_update_conda false && \
conda config --system --set show_channel_urls true && \
if [ ${PYTHON_VERSION} != "default" ]; then mamba install --quiet --yes python="${PYTHON_VERSION}"; fi && \
mamba list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> "${CONDA_DIR}/conda-meta/pinned" && \
# Using conda to update all packages: https://github.com/mamba-org/mamba/issues/1092
conda update --all --quiet --yes && \
conda clean --all -f -y && \
rm -rf "/home/${NB_USER}/.cache/yarn"
# install R dependencies
RUN mamba install --quiet --yes \
'r-base' \
'r-cairo' \
'r-ragg' \
'r-ggrastr' \
'r-arrow' \
'r-biocmanager' \
'r-foreign' \
'r-ggpubr' \
'r-rmarkdown' \
'r-seurat' \
'r-tidyverse' \
'r-viridis' \
'r-ellipsis' \
'r-processx' && \
mamba clean --all -f -y
WORKDIR /
RUN git clone https://github.com/interactivereport/cellxgene_VIP.git
WORKDIR /cellxgene_VIP
# install Python depdendencies
ADD vip_requirements.txt /cellxgene_VIP/vip_requirements.txt
RUN pip install -r /cellxgene_VIP/vip_requirements.txt
RUN bash config.sh
# install cellxgene Python dependencies if any
RUN pip install -r /cellxgene_VIP/cellxgene/server/requirements.txt
The following is the vip_requirements.txt
used in the Dockerfile, extracted from the VIP.yml in this repo.
aniso8601==8.1.1
anndata==0.7.4
attrs==20.3.0
batchglm==0.7.4
bleach==3.2.3
boto3==1.14.39
botocore==1.17.44
brotli==1.0.9
cffi==1.14.4
chardet==4.0.0
click==7.1.2
cycler==0.10.0
dask==2021.1.1
defusedxml==0.6.0
diffxpy==0.7.4
docutils==0.15.2
entrypoints==0.3
fastobo==0.9.3
flask==1.1.2
flask-compress==1.8.0
flask-cors==3.0.10
flask-restful==0.3.8
flask-server-timing==0.1.2
flask-talisman==0.7.0
flatbuffers==1.12
flatten-dict==0.3.0
fsspec==0.7.4
get-version==2.1
gunicorn==20.0.4
h5py==2.10.0
idna==2.10
iniconfig==1.1.1
itsdangerous==1.1.0
jinja2==2.11.2
jmespath==0.10.0
joblib==1.0.0
jsonschema==3.2.0
jupytext==1.6.0
kiwisolver==1.3.1
legacy-api-wrap==1.2
llvmlite==0.35.0
markdown-it-py==0.5.8
markupsafe==1.1.1
matplotlib==3.3.4
mistune==0.8.4
natsort==7.1.1
nbconvert==5.6.1
nbformat==5.1.2
networkx==2.5
numba==0.52.0
numexpr==2.7.2
numpy==1.19.5
packaging==20.9
pandas==1.2.1
pandocfilters==1.4.3
pathlib2==2.3.5
patsy==0.5.1
pillow==8.1.0
plotly==4.8.1
pluggy==0.13.1
py==1.10.0
pyarrow==1.0.1
pycparser==2.20
pyparsing==2.4.7
pyrsistent==0.17.3
pytest==6.2.2
pytz==2020.5
pyyaml==5.4.1
requests==2.25.1
retrying==1.3.3
rpy2==3.3.5
s3fs==0.4.2
s3transfer==0.3.4
scanpy==1.6.1
scikit-learn==0.24.1
scipy==1.6.0
seaborn==0.11.1
setuptools-scm==5.0.1
sinfo==0.3.1
sparse==0.11.2
sqlalchemy==1.3.22
statsmodels==0.12.1
stdlib-list==0.8.0
tables==3.6.1
testpath==0.4.4
threadpoolctl==2.1.0
tiledb==0.7.7
toml==0.10.2
toolz==0.11.1
tqdm==4.56.0
tzlocal==2.1
umap-learn==0.4.6
urllib3==1.25.11
webencodings==0.5.1
werkzeug==1.0.1