Missing package `charset_normalizers`

[goudbor]

I cloned repo to check fresh install of instructions. pipenv skips many packages because of "environment mismatch".

Steps to Reproduce

1. Clone repo and follow setup instructions

Expected behavior Successful environment setup

Actual behavior python -m pipenv install exits successfully but skips many packages. Running uvicorn opendevin.server.listen:app --port 3000 generates error

Traceback (most recent call last):
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/main.py", line 409, in main
    run(
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/main.py", line 575, in run
    server.run()
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/importer.py", line 22, in import_from_string
    raise exc from None
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/anon/sandbox/OpenDevin/opendevin/server/listen.py", line 1, in <module>
    from opendevin.server.session import Session
  File "/home/anon/sandbox/OpenDevin/opendevin/server/session.py", line 8, in <module>
    from opendevin.action import (
  File "/home/anon/sandbox/OpenDevin/opendevin/action/__init__.py", line 3, in <module>
    from .browse import BrowseURLAction
  File "/home/anon/sandbox/OpenDevin/opendevin/action/browse.py", line 1, in <module>
    import requests
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/requests/exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "/home/anon/.local/share/virtualenvs/OpenDevin-xSXWottn/lib/python3.11/site-packages/requests/compat.py", line 13, in <module>
    import charset_normalizer as chardet
ModuleNotFoundError: No module named 'charset_normalizer'

Additional context Looking through pipenv logs it says it skips charset-normalizer due to environment mismatch

Ignoring charset-normalizer: markers 'python_full_version >= "3.7.0"' don't match your environment 

pipenv --support shows

Pipenv version: `'2023.12.1'`

Pipenv location: `'/home/george/.local/lib/python3.11/site-packages/pipenv'`

Python location: `'/usr/bin/python3.11'`

OS Name: `'posix'`

User pip version: `'23.3.2'`

user Python installations found:


PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.0c1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.5.0-26-generic',
 'platform_system': 'Linux',
 'platform_version': '#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 '
                     '10:22:43 UTC 2',
 'python_full_version': '3.11.0rc1',
 'python_version': '3.11',
 'sys_platform': 'linux'}

So my python version satisfies the constraint but it still skips the package install as well as other critical packages. Also it seems that the python being used is my system version in /usr/bin and not the virtual environment.

Manually installing charset-normalizer does not work either. I was the one who originally changed the project to use pipenv but I think we should abandon it and immediately revert back to using a requirements.txt along with venv. pipenv does not seem to make builds more reproducible. After reverting we can think about moving to poetry or something else.

[omutist]

I installed 3.11 on Ubuntu 22.04 just couple of hours ago and everything was OK - I used requirements.txt generated by pipenv. The only step missing in your description was installation of pip for 3.11

python3.11 -m venv .venv --without-pip
source .venv/bin/activate
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

[808vita]

I had similar issues when I was trying out another opensource project. I use miniconda and initially faced the same issue of system python being used. To fix that after activating the conda env I had to do conda install pip and then did some few other things.

[goudbor]

@808vita yeah i'm sure i could play around and get it working but i don't expect most users to and it creates unnecessary friction in the setup. So i think it would be better just to revert to requirements.txt for now.

[goudbor]

@omutist If users are having to generate a requirements.txt with pipenv and then use venv it's better to just use a requirements.txt and get rid of pipenv.

[enyst]

Also it seems that the python being used is my system version in /usr/bin and not the virtual environment.

I think I noticed something like that too, although with a different setup (with brew in the mix for maximum fun). I'm sure it's fixable but I would agree that something about pipenv is a bit too counter-intuitive, unlike good ole' pip+venv.

For the record, opendevin works just fine with venv on a requirements.txt hacked from the former venv + pipenv lock file. There's a PR for poetry here: https://github.com/OpenDevin/OpenDevin/pull/378

There's also a thread on this in Slack. Are you there?

[goudbor]

@enyst Not on Slack, will fill out the form

[goudbor]

Submitted form. Doing Sunday family stuff but can be on later today (PCT)

[andrewparry]

The pipfile.lock may have some incorrect python_versions set. E.g. shouldn't this be <='3.11'

       "async-timeout": {
            "hashes": [
                "sha256:4640d96be84d82d02ed59ea2b7105a0f7b33abe8703703cd0ab0bf87c427522f",
                "sha256:7405140ff1230c310e51dc27b3145b9092d659ce68ff733fb0cefe3ee42be028"
            ],
            "markers": "python_version < '3.11'",
            "version": "==4.0.3"
         }

or python_full_version maybe should be python_version

       "llama-index": {
            "hashes": [
                "sha256:25a3f8c9f63fbdaaf3d760ce59c5dad8afc5cf40431e9e6a28e02b92a820d450",
                "sha256:5a6036bcb2449277ede3244cfa1b49d9fab5dba30ad8b212c2df92449cfa7d48"
            ],
            "index": "pypi",
            "markers": "python_version < '4.0' and python_full_version >= '3.8.1'",
            "version": "==0.10.26"
        }

I manually updated and everything installed & unicorn started

[goudbor]

@andrewparry I'm not sure. There were many packages that were skipped when I tried installing with 3.11. The first one that came up was charset-normalizers. The marker for that package is

"markers": "python_full_version >= '3.7.0'",

which should be satisfied?

[andrewparry]

@jojeyh In this case I changed python_full_version to python_version to make it work.

"markers": "python_version >= '3.7.0'",

I could not make python_full_version work

[rbren]

Going to close this one now that we have poetry. Feel free to open a new issue if you're still having problems