Switch to Poetry

[anasdorbani]

• Related to this: #374

  • Create Project Setup file
  • Update Readme
  • Update .gitignore
  • Remove Pipfile
  • Fix mypy issue for toml

[rbren]

Thanks for this!

For others' context: there's an active discussion in Slack as to whether this is the right path forward.

[goudbor]

@dorbanianas venv is not necessary with poetry, it handles the virtual environment. The environment is stored in {cache_dir}/virtualenvs, same as with pipenv. The environment is activated by running poetry shell

https://python-poetry.org/docs/basic-usage

[anasdorbani]

@jojeyh Yeah I forgot that thanks for the info

[rbren]

There's general consensus that we should make this switch. @dorbanianas can you rebase (or maybe recreate) this PR? Lots of new dependencies

[anasdorbani]

You can check it now I fixed the dependencies versions and deleted the lock files, in the future we should take in consideration the pyproject.toml and makefile updates on each commit

[rbren]

This LGTM! One comment on removing the langchain deps

[anasdorbani]

@rbren can you pls check the PR now

[foragerr]

What's the outcome of the Slack discussion? I filled the form to get access to slack, but I didn't hear back yet :( I'd love to see this repo converted to poetry.

[anasdorbani]

@foragerr we are almost there I need just someone to merge this PR

[rbren]

I'm currently stuck on this error:

Cannot install pymupdfb.

Full logs:

✖ make build
Building project...
Pulling Docker image...
Using default tag: latest
latest: Pulling from opendevin/sandbox
Digest: sha256:486e3da1d7c8ec54a2e3269071eed9b91bc75b82ca5d1ac1760d33883c39e575
Status: Image is up to date for ghcr.io/opendevin/sandbox:latest
ghcr.io/opendevin/sandbox:latest
Installing Python dependencies...
Retrieving Poetry metadata

The latest version (1.8.2) is already installed.
Updating dependencies
Resolving dependencies... (6.3s)

Package operations: 78 installs, 0 updates, 0 removals

  - Installing pymupdfb (1.24.1): Failed

  RuntimeError

  Unable to find installation candidates for pymupdfb (1.24.1)

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:74 in choose_for
       70│
       71│             links.append(link)
       72│
       73│         if not links:
    →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       75│
       76│         # Get the best link
       77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       78│

Cannot install pymupdfb.

make: *** [Makefile:19: build] Error 1

[rbren]

More logs. I'm on Ubuntu ARM--seems like that might be the issue

✖ poetry add pymupdfb -vvvv
...
Package operations: 78 installs, 0 updates, 0 removals, 115 skipped

  - Installing pymupdfb (1.24.1): Pending...
Skipping wheel PyMuPDFb-1.24.1-py3-none-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel PyMuPDFb-1.24.1-py3-none-macosx_11_0_arm64.whl as this is not supported by the current environment
Skipping wheel PyMuPDFb-1.24.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl as this is not supported by the current environment
Skipping wheel PyMuPDFb-1.24.1-py3-none-win32.whl as this is not supported by the current environment
Skipping wheel PyMuPDFb-1.24.1-py3-none-win_amd64.whl as this is not supported by the current environment
  - Installing pymupdfb (1.24.1): Failed

  Stack trace:

  5  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:281 in _execute_operation
      279│
      280│             try:
    → 281│                 result = self._do_execute_operation(operation)
      282│             except EnvCommandError as e:
      283│                 if e.e.returncode == -2:

  4  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:391 in _do_execute_operation
      389│             return 0
      390│
    → 391│         result: int = getattr(self, f"_execute_{method}")(operation)
      392│
      393│         if result != 0:

  3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:516 in _execute_install
      514│
      515│     def _execute_install(self, operation: Install | Update) -> int:
    → 516│         status_code = self._install(operation)
      517│
      518│         self._save_url_reference(operation)

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:554 in _install
      552│             archive = self._download_link(operation, Link(package.source_url))
      553│         else:
    → 554│             archive = self._download(operation)
      555│
      556│         operation_message = self.get_operation_message(operation)

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:732 in _download
      730│
      731│     def _download(self, operation: Install | Update) -> Path:
    → 732│         link = self._chooser.choose_for(operation.package)
      733│
      734│         if link.yanked:

  RuntimeError

  Unable to find installation candidates for pymupdfb (1.24.1)

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:74 in choose_for
       70│
       71│             links.append(link)
       72│
       73│         if not links:
    →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       75│
       76│         # Get the best link
       77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       78│

Cannot install pymupdfb.

[anasdorbani]

@rbren I guess you don't have python 3.11 you are using python 3.10

[anasdorbani]

Does pymupdfb something in the dependencies?

[anasdorbani]

Try to remove potry.lock if exists there

[rbren]

❯ python --version
Python 3.12.2

But there is 3.10 on this machine

[rbren]

Looks like it's a llamaindex dep

✖ poetry add pymupdf==1.22.5

Updating dependencies
Resolving dependencies... (12.1s)Source (pytorch): Authorization error accessing https://download.pytorch.org/whl/cpu/llama-index-readers-file/
Resolving dependencies... (12.5s)Source (pytorch): Authorization error accessing https://download.pytorch.org/whl/cpu/llama-index/
Resolving dependencies... (12.6s)

    Because llama-index (0.10.25) depends on llama-index-readers-file (>=0.1.4,<0.2.0)
 and no versions of llama-index match >0.10.25,<0.10.26 || >0.10.26,<0.11.0, llama-index (>=0.10.25,<0.10.26 || >0.10.26,<0.11.0) requires llama-index-readers-file (>=0.1.4,<0.2.0).
(1) So, because llama-index (0.10.26) depends on llama-index-readers-file (>=0.1.4,<0.2.0), llama-index (>=0.10.25,<0.11.0) requires llama-index-readers-file (>=0.1.4,<0.2.0).

    Because no versions of llama-index-readers-file match >0.1.4,<0.1.5 || >0.1.5,<0.1.6 || >0.1.6,<0.1.7 || >0.1.7,<0.1.8 || >0.1.8,<0.1.9 || >0.1.9,<0.1.11 || >0.1.11,<0.1.12 || >0.1.12,<0.1.13 || >0.1.13,<0.2.0
 and llama-index-readers-file (0.1.4) depends on pymupdf (>=1.23.21,<2.0.0), llama-index-readers-file (>=0.1.4,<0.1.5 || >0.1.5,<0.1.6 || >0.1.6,<0.1.7 || >0.1.7,<0.1.8 || >0.1.8,<0.1.9 || >0.1.9,<0.1.11 || >0.1.11,<0.1.12 || >0.1.12,<0.1.13 || >0.1.13,<0.2.0) requires pymupdf (>=1.23.21,<2.0.0).
    And because llama-index-readers-file (0.1.5) depends on pymupdf (>=1.23.21,<2.0.0)
 and llama-index-readers-file (0.1.6) depends on pymupdf (>=1.23.21,<2.0.0), llama-index-readers-file (>=0.1.4,<0.1.7 || >0.1.7,<0.1.8 || >0.1.8,<0.1.9 || >0.1.9,<0.1.11 || >0.1.11,<0.1.12 || >0.1.12,<0.1.13 || >0.1.13,<0.2.0) requires pymupdf (>=1.23.21,<2.0.0).
    And because llama-index-readers-file (0.1.7) depends on pymupdf (>=1.23.21,<2.0.0)
 and llama-index-readers-file (0.1.8) depends on pymupdf (>=1.23.21,<2.0.0), llama-index-readers-file (>=0.1.4,<0.1.9 || >0.1.9,<0.1.11 || >0.1.11,<0.1.12 || >0.1.12,<0.1.13 || >0.1.13,<0.2.0) requires pymupdf (>=1.23.21,<2.0.0).
    And because llama-index-readers-file (0.1.9) depends on pymupdf (>=1.23.21,<2.0.0)
 and llama-index-readers-file (0.1.11) depends on pymupdf (>=1.23.21,<2.0.0), llama-index-readers-file (>=0.1.4,<0.1.12 || >0.1.12,<0.1.13 || >0.1.13,<0.2.0) requires pymupdf (>=1.23.21,<2.0.0).
    And because llama-index-readers-file (0.1.12) depends on pymupdf (>=1.23.21,<2.0.0)
 and llama-index-readers-file (0.1.13) depends on pymupdf (>=1.23.21,<2.0.0), llama-index-readers-file (>=0.1.4,<0.2.0) requires pymupdf (>=1.23.21,<2.0.0).
    And because llama-index (>=0.10.25,<0.11.0) requires llama-index-readers-file (>=0.1.4,<0.2.0) (1), llama-index (>=0.10.25,<0.11.0) requires pymupdf (>=1.23.21,<2.0.0)
    So, because opendevin depends on both pymupdf (1.22.5) and llama-index (^0.10.25), version solving failed.

[rbren]

Maybe related: https://github.com/pymupdf/PyMuPDF/issues/2613

I need to go to bed for now, but will take a look again tomorrow

[anasdorbani]

Yes me too it's 4AM here 😅 good night

[enyst]

Just a heads-up. On MacBook, new clean directory, following the readme exactly, make build.

  - Installing pypdf (4.1.0)
  - Installing pypika (0.48.9)
  - Installing scikit-learn (1.4.1.post1)
  - Installing striprtf (0.0.26)
  - Installing torch (2.2.2+cpu): Failed

  RuntimeError

  Unable to find installation candidates for torch (2.2.2+cpu)
  Unable to find installation candidates for torch (2.2.2+cpu)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.12/site-packages/poetry/installation/chooser.py:74 in choose_for
       70│ 
       71│             links.append(link)
       72│ 
       73│         if not links:
    →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       75│ 
       76│         # Get the best link
       77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       78│ 

Cannot install torch.

[foragerr]

FWIW, I'm having trouble with pymupdfb, even without poetry. The current head of main doesn't do pipenv install or make build cleanly.

I tried bouncing between python 3.10, 3.11 and 3.12, didn't help. I'm on an M1 mac.

[rbren]

[anasdorbani]

@foragerr can you try this pls?

brew install mupdf swig freetype
pip install --upgrade pymupdf

[foragerr]

I installed pymupdfb with a pip install, and then ran into the next M1 incompatibility. nvidia-nvjitlink-cu12 does not have a mac compatible distribution apparently?

I found more success with removing playwright as a dependency, then pipfile install runs clean, then running pip install playwright

[foragerr]

BTW, @dorbanianas your PR does not contain a poetry.lock file?

[enyst]

Please see also about pymupdfb: https://github.com/OpenDevin/OpenDevin/pull/625#issuecomment-2035194367

[anasdorbani]

@rbren what do you think about it? have you any suggestions

[anasdorbani]

@rbren @neubig I guess we should prioritize this switch and trying to test the execution because we received so many issues related to the project setup especially at this moment where the project is going so viral so having a a good setup is mendatory.

[rbren]

the dep is fixed for me on main. Can you rebase?

[foragerr]

@dorbanianas shouldn't you be including a poetry.lock file?

[anasdorbani]

@foragerr I'm not sure about that but if it seems necessary to you I can include it