gpt-engineer Docker build is broken

Expected Behavior

After running docker build --rm -t gpt-engineer -f docker/Dockerfile . I should have a runnable container.

Current Behavior

Unable to build the container due to PyArrow and CMake issues.

Failure Information

Failure Logs

  creating /tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241/build/temp.linux-x86_64-cpython-311
43.03       -- Running cmake for PyArrow
43.03       cmake -DCMAKE_INSTALL_PREFIX=/tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241/build/lib.linux-x86_64-cpython-311/pyarrow -DPYTHON_EXECUTABLE=/usr/local/bin/python -DPython3_EXECUTABLE=/usr/local/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_AZURE=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241
43.03       error: command 'cmake' failed: No such file or directory
43.03       [end of output]
43.03
43.03   note: This error originates from a subprocess, and is likely not a problem with pip.
43.03   ERROR: Failed building wheel for pyarrow

Apr 28 '24 10:04 t6tv8e

command 'cmake' failed: No such file or directory

May 02 '24 01:05 Thinking80s

Hi @t6tv8e - I think the issue is related to the fact that cmake utility is not installed or not available in the system's PATH where you are trying to install the pyarrow package. You might want to check their docs.

Or, depending on your OS, install it along these lines (this should work for Ubuntu or Debian-based stuff:

sudo apt-get update
sudo apt-get install cmake

May 02 '24 07:05 viborc

I'm on macos and I have cmake on my machine

May 02 '24 07:05 t6tv8e

@t6tv8e, can you try running this command in your Terminal and share the output/screenshot here:

cmake --version

May 02 '24 07:05 viborc

Screenshot 2024-05-02 at 09 55 24

May 02 '24 07:05 t6tv8e

Let's check if it's configured properly.

mkdir ~/cmake-test
cd ~/cmake-test
echo "cmake_minimum_required(VERSION 3.10)" > CMakeLists.txt
echo "project(HelloCMake)" >> CMakeLists.txt
cmake .

This should check if it is functioning correctly by creating a temporary project directory and attempting to configure a simple project.

If that fails, you might want to consider reinstalling it using homebrew or whatever you usually use like this:

brew install cmake

May 02 '24 07:05 viborc

Screenshot 2024-05-02 at 10 10 51

May 02 '24 08:05 t6tv8e

Screenshot 2024-05-02 at 10 11 49

May 02 '24 08:05 t6tv8e

Hm... what's the output of your PATH?

echo $PATH

May 02 '24 08:05 viborc

That is: /Users/t6t/.pyenv/shims:/Users/t6t/.nvm/versions/node/v20.12.1/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/t6t/.dotnet/tools

May 02 '24 08:05 t6tv8e

Hm... can you do this please:

which cmake

May 02 '24 08:05 viborc

/opt/homebrew/bin/cmake

May 02 '24 08:05 t6tv8e

Okay, I'll need to check this later today with the team. I'm unsure what the issue might be. Maybe you can try reinstalling the cmake and see if that makes any difference.

May 02 '24 08:05 viborc

Okay, thanks for your time already in any case

May 02 '24 08:05 t6tv8e

Of course, we are always trying to help!

May 02 '24 08:05 viborc

The issue is with cmake missing in the container, not the host system.

Seems the Dockerfile needs:

RUN apk add --update build-base or similar.

May 02 '24 11:05 ErikBjare

Yeah, I've been tinkering with the Dockerfile.

FROM python:3.10-alpine AS builder

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \
    build-base

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.10-alpine

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \ 
    build-base

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["sh", "/app/entrypoint.sh"]

May 02 '24 11:05 t6tv8e

Didn't work

May 02 '24 11:05 t6tv8e

Identical error?

May 02 '24 11:05 ErikBjare

Yes it's the same error...

May 02 '24 12:05 t6tv8e

Hey @t6tv8e, we just had our technical planning meeting and discussed this. One of our colleagues tried adding cmake to a Docker image, but that didn't solve the problem. We have a couple of other options we'll try out, but we'll keep you updated!

@ErikBjare, thanks for helping with this; I'm super thankful for checking on this, especially while you are away! 🙏

May 02 '24 17:05 viborc

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:

FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

May 02 '24 18:05 zigabrencic

That sounds good. Let's see what @t6tv8e says, and we can then figure out the next steps. Thanks for a quick turnaround, @zigabrencic! 💪

May 02 '24 18:05 viborc

@zigabrencic @viborc

That worked 👌

May 02 '24 23:05 t6tv8e

Perfect! We are now looking into testing this on Linux, Mac, and Windows and we'll see how to fix it for everyone. I'll leave this issue open for now just as a reminder to @zigabrencic and me to follow-up on this for other platforms.

Glad it worked for you @t6tv8e! Thanks, @zigabrencic and @ErikBjare for helping out!

May 03 '24 13:05 viborc

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:
FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]
And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

@zigabrencic The no-cache flag is not the source of the issue. While pyarrow is supported on Alpine, it lacks a wheel. In other words, you need to build it from source.

I notice that the Dockerfile in #1144 will be reverted; use this one instead:

# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

The image size would still be reduced

And on the other hand, Docker containers are designed to run with an unprivileged user by default, and running commands with sudo, for reference RUN sudo pip install -e . , or as the root user can lead to potential security issues and unexpected behavior.

May 17 '24 21:05 k1lgor

Hey

Thank you for your suggestions.

Yes you are correct @k1lgor the cmake dependency broke this. I misstated this before.

Tried your proposal and:

1.) Image on Mac OS is still 1.43 GB's of size.
2.) Fails with: FileNotFoundError: [Errno 2] No such file or directory: 'git' once gpte engineer is executed.` Despite the fact that git is installed.

As a result I propose we keep the simpler version as implemented here.

The cross platform size benefit in my opinion isn't large enough to add the proposed complexity.

I made some updates to #1144:

change the RUN sudo pip part to RUN pip => per @k1lgor proposal.
python:3-11-slim to python:3-12-slim
Added the line: pip install -U langchain-community => We might have to add this one elsewhere.

If there are no objections I'll open the PR again.

May 18 '24 07:05 zigabrencic

Hey @zigabrencic ,

I believe I may be able to locate the issue. tk tcl curl git are installed in /usr/bin/ since they are dpkg packages. It is necessary to add one COPY.

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
+ COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

And in addition, I have added langchain-community = "0.2.0" into pyproject.toml. Could you test it on your Mac?

May 18 '24 10:05 k1lgor

Hey @k1lgor

Sure I can test it just please let me know which branch?

Or if you can provide the full docker file so I test the right thing :)

May 18 '24 10:05 zigabrencic

Here are the files. In the pyproject.toml I only added langchain-community ="0.2.0" and in Dockerfile - COPY --from=builder /usr/bin /usr/bin

Dockerifle

```bash
# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

pyproject.toml

[tool.poetry]
name = "gpt-engineer"
version = "0.3.0"
description = "Specify what you want it to build, the AI asks for clarification, and then builds it."
authors = ["Anton Osika <[email protected]>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/gpt-engineer-org/gpt-engineer"
repository = "https://github.com/gpt-engineer-org/gpt-engineer"
documentation = "https://gpt-engineer.readthedocs.io/en/latest/"
classifiers = [
  "Development Status :: 4 - Beta",
  "Topic :: Scientific/Engineering :: Artificial Intelligence",
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
openai = "^1.0"
termcolor = "2.3.0"
typer = ">=0.3.2"
rudder-sdk-python = ">=2.0.2"
dataclasses-json = "0.5.7"
tiktoken = ">=0.0.4"
tabulate = "0.9.0"
python-dotenv = ">=0.21.0"
langchain = ">=0.1.2"
langchain_openai = "*"
toml = ">=0.10.2"
tomlkit = "^0.12.4"
pyperclip = "^1.8.2"
langchain-anthropic = "^0.1.1"
regex = "^2023.12.25"
pillow = "^10.2.0"
datasets = "^2.17.1"
black = "23.3.0"
langchain-community ="0.2.0"

[tool.poetry.group.dev.dependencies]
pytest = ">=7.3.1"
pytest-cov = "^4.1.0"
mypy = "1.3.0"
ruff = ">=0.0.272"
pre-commit = "3.3.3"
tox = ">=3.0.0"

[tool.poetry.group.docs.dependencies]
autodoc_pydantic = ">=1.8.0"
myst_parser = ">=0.18.1"
nbsphinx = ">=0.8.9"
sphinx = ">=5.0.0"
sphinx-autobuild = ">=2021.3.14"
sphinx_book_theme = ">=0.3.3"
sphinx_rtd_theme = ">=1.0.0"
sphinx-typlog-theme = ">=0.8.0"
toml = ">=0.10.2"
myst-nb = ">=0.17.1"
linkchecker = ">=10.2.1"
sphinx-copybutton = ">=0.5.1"
markdown-include = ">=0.6.0"
sphinx_copybutton = ">=0.5.2"

[tool.poetry.scripts]
gpt-engineer = 'gpt_engineer.applications.cli.main:app'
ge = 'gpt_engineer.applications.cli.main:app'
gpte = 'gpt_engineer.applications.cli.main:app'
bench = 'gpt_engineer.benchmark.__main__:app'
gpte_test_application = 'tests.caching_main:app'

[tool.poetry.extras]
test = ["pytest", "pytest-cov"]
doc = ["autodoc_pydantic", "myst_parser", "nbsphinx", "sphinx", "sphinx-autobuild", "sphinx_book_theme", "sphinx_rtd_theme", "sphinx-typlog-theme", "myst-nb", "linkchecker", "sphinx-copybutton", "markdown-include", "sphinx_copybutton"]

[tool.ruff]
select = ["F", "E", "W", "I001"]
show-fixes = false
target-version = "py310"
task-tags = ["TODO", "FIXME"]
extend-ignore = ["E501", "E722"]

[tool.black]
target-version = ["py310"]

[tool.ruff.isort]
known-first-party = []
known-third-party = []
section-order = [
    "future",
    "standard-library",
    "third-party",
    "first-party",
    "local-folder",
]
combine-as-imports = true
split-on-trailing-comma = false
lines-between-types = 1

[tool.pytest.ini_options]
markers = [
    "requires_key: marks tests as requiring access to a valid OPENAI_API_KEY (deselect with '-m \"not requires_key\"')",
]

May 18 '24 11:05 k1lgor

gpt-engineer gpt-engineer copied to clipboard

Docker build is broken

Expected Behavior

Current Behavior

Failure Information

Failure Logs

gpt-engineer
gpt-engineer copied to clipboard