docker dev and test environment

[blob42]

implements feature for #1031

TODO:

• [ ~Use pyenv for setting custom python versions and virtualenvs*~
• [x] Separate makefile for docker
  • [x] hidden from users who don't have docker installed
• [x] Virtualized python env
• [x] all python env commands available for container
  • example launch jupyter notebook docker run IMG jupyter notebook
• [x] Setup dev-test- repl workflow
  • [x] ipython/notebook from within container with exported ports
• [x] Documentation
  • [x] make docker commands
  • [x] docker-compose
  • [x] common tasks examples
• [x] using docker for testing / linting

_* ~pyenv might be necessary to make sure any python version can be installed. Given the multitude of dependencies some users might have issues with certain packages or missing libraries. Also useful for testing under different pythons.~

Abondonted the poetry/pyenv way and used a pure python based container. Using virtualenvs inside docker is a controversial topic and adds a lot of complexity. long discussion here

USAGE:

• Use make docker or just type make for a list of commands
• All commands that in the python env are available by default in the container. For example:

# run jupyter notebook
docker run --rm -it IMG jupyter notebook

# run ipython
docker run --rm -it IMG ipython

# start web server
docker run --rm -p 8888:8888 IMG python -m http.server 8888

• Running make docker.run or make docker.build will automatically rebuild the image if needed when code is changed, only the package is reinstalled. Dependencies are only built once using docker builder cache.

• linting can happen with parallel jobs using make -j4 docker.lint

NOTE: using docker for sandboxing untrusted code exection is tracked in #1031

[blob42]

@hwchase17 I moved the discussion on sandboxing to #1031 not to keep this PR blocked and to separate concerns between development images and security ones.

This should be good to merge, minus some extra documentation I will add later. I was thinking to add a docker.md under the docs folder that can be built to html.

[blob42]

@hwchase17 feel free to merge this one. It provides all what is needed for developing inside docker containers. The makefile is optional and only included if docker is detected on the host. It should not interfere with the current build process.

My work on #1266 depends on this one.

[pruksmhc]

Hi @blob42 thanks for implementing this! I tried to build the dev Docker image using make docker. However, when I'm in the container, it seems to not contain any libraries. Do you know what might be the issue? `(base) Tests-MacBook-Pro langchain-blob % make docker Docker image: langchain/dev:a9b8cf0 The dependencies have been installed in the current shell. There is no virtualenv or a need for poetry inside the container. Running the command make docker.run at the root directory of the project will build the container the first time. On the next runs, it will use the cached image or rebuild it if a change is made on the source code. langchain 0.0.90 Python 3.11.2 lchain@49f6795d509f:~$ python3 Python 3.11.2 (main, Feb 9 2023, 05:12:33) [GCC 10.2.1 20210110] on linux Type "help", "copyright", "credits" or "license" for more information. python3

import torch Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'torch' `

What is going wrong here?

[blob42]

@pruksmhc The issue is the missing poetry extras in the built image. I did not include all the extras. The docker image is built with poetry export --with dev dependencies, including all the extras generated a lot of conflicts. My thinking was that users would install extra packages for their specific needs.

I will try to update the Dockerfile to include all the extras, it will take me some more time to solve all the conflicts (see below)

ERROR: Ignored the following versions that require a different python version:
0.11.10 Requires-Python >=3.7,<=3.11; 0.28.0 Requires-Python >=3.7, <3.11; 1.0.0
Requires-Python >=3.7,<=3.11; 1.0.1 Requires-Python >=3.7,<=3.11; 1.0.2
Requires-Python >=3.7,<=3.11; 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3
Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5
Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 1.6.2
Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0
Requires-Python >=3.7,<3.10; 1.7.0rc1 Requires-Python >=3.7,<3.10; 1.7.0rc2
Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.7.2
Requires-Python >=3.7,<3.11; 1.7.3 Requires-Python >=3.7,<3.11; 1.8.0
Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2
Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4
Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11
ERROR: Could not find a version that satisfies the requirement
tensorflow-text==2.11.0 (from versions: 2.12.0rc0)
ERROR: No matching distribution found for tensorflow-text==2.11.0

EDIT: You can install any package inside the container with the usual pip workflow though,

[blob42]

@pruksmhc it should be fixed now. I had to downgrade to python 3.10 to install all extras.

There is still the possibility to build a slimmer image with dev only deps by modifying the docker/.env file.

[pruksmhc]

Thank you!

[eyurtsev]

Amazing!

[blob42]

@eyurtsev I merged the conflicts, however since someone already made docker_tests rule I kept it since it uses an image for tests only. This PR also includes a docker test that uses the dev image. Not sure which one to keep.

[leo-gan]

@eyurtsev It is you to decide on Closing.

[efriis]

Hey @blob42! This predates me, and I think some of the docker infra might have evolved since this was open! If you feel there's parts of this that still needs to land, feel free to reopen against the current state of the repo :)