Cog compatible image/container spec / allow base image in cog.yaml

[DeNeutoy]

Hello!

I know that cog is aimed at research projects/researchers not super familiar with docker. However, I am investigating deploying models on replicate for the company I work for (i.e private models), and we already have a fully containerized workflow, which works with GPUs. It would be great if I could specify a parent image in the build section of cog.yaml.

Perhaps this already works, and it is only the api specification of having a Predictor and associated server entrypoint in a given docker image that makes it "replicate" compatible, although I somehow doubt this is the case.

My ideal workflow would be:

• I have a prebuilt docker image
• I write a predictor, and create a new docker image with the predictor as the entrypoint
• I can push this image to replicate and run it

Perhaps this is hard! Let me know if you have any suggestions for an ideal approach here. Thanks!

[technillogue]

We don't support this, as cog requires a lot of assumptions about running on ubuntu. However, as of 0.9.0-beta5, you can write the entire dockerfile yourself and use --dockerfile to pass it in. cog debug can be helpful for this. This is not officially supported; you're on your own for any errors. You should have cog.yaml in the workdir, have python3 -m cog.server.http and python3 -m cog.command.openapi_schema working, and have pip install cog==<version> be a valid command.

[ynie]

@technillogue hello, i'm on cog version 0.9.0-beta1. and I can't get --dockerfile working.

Validating model schema...
ⅹ Model schema is invalid: value of openapi must be a non-empty string
ⅹ 
ⅹ {}

running python3 -m cog.command.openapi_schema does output the schema.. did I miss anything? Thanks!

[ynie]

It seems like this will work: cog push r8.im/ynie/alteraspect --dockerfile Dockerfile.replicate --openapi-schema openapi_schema but it will stuck at starting state.

[technillogue]

can you send me your dockerfile? it needs to match the output of cog debug fairly closely. pip install -U cog and python3 -m cog.server.http both have to work for your model to start. it's also worth checking if your container will boot at all on your own machine, or if you have some error

also, currently please use cog 0.9.0-beta10

[ynie]

@technillogue Ok, I was on cog version 0.9.0-beta11 (built 2023-11-28T17:38:26Z). Maybe that's the issue?

#syntax=docker/dockerfile:1.4
FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib64:/usr/local/nvidia/bin
RUN --mount=type=cache,target=/var/cache/apt set -eux; \
apt-get update -qq; \
apt-get install -qqy --no-install-recommends curl; \
rm -rf /var/lib/apt/lists/*; \
TINI_VERSION=v0.19.0; \
TINI_ARCH="$(dpkg --print-architecture)"; \
curl -sSL -o /sbin/tini "https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-${TINI_ARCH}"; \
chmod +x /sbin/tini
ENTRYPOINT ["/sbin/tini", "--"]
ENV PATH="/root/.pyenv/shims:/root/.pyenv/bin:$PATH"
RUN --mount=type=cache,target=/var/cache/apt apt-get update -qq && apt-get install -qqy --no-install-recommends \
	make \
	build-essential \
	libssl-dev \
	zlib1g-dev \
	libbz2-dev \
	libreadline-dev \
	libsqlite3-dev \
	wget \
	curl \
	llvm \
	libncurses5-dev \
	libncursesw5-dev \
	xz-utils \
	tk-dev \
	libffi-dev \
	liblzma-dev \
	git \
	ca-certificates \
	&& rm -rf /var/lib/apt/lists/*
RUN curl -s -S -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash && \
	git clone https://github.com/momo-lab/pyenv-install-latest.git "$(pyenv root)"/plugins/pyenv-install-latest && \
	pyenv install-latest "3.11" && \
	pyenv global $(pyenv install-latest --print "3.11") && \
	pip install "wheel<1"
COPY replicate_server/cog-0.0.1.dev-py3-none-any.whl /tmp/cog-0.0.1.dev-py3-none-any.whl
RUN --mount=type=cache,target=/root/.cache/pip pip install /tmp/cog-0.0.1.dev-py3-none-any.whl
RUN --mount=type=cache,target=/var/cache/apt apt-get update -qq && apt-get install -qqy libgl1-mesa-glx && rm -rf /var/lib/apt/lists/*
COPY requirements.replicate.txt /tmp/requirements.txt
RUN --mount=type=cache,target=/root/.cache/pip pip install -r /tmp/requirements.txt
WORKDIR /src
COPY checkpoints /src/checkpoints
EXPOSE 5000
CMD ["python", "-m", "cog.server.http"]
COPY [^checkpoints]. /src