[Install issue]: Impossible to install on AWS (t3.small & t3.medium)

[ezesculli]

What happened?

When trying to include chromadb dependency on requirements.txt and deploy python app to AWS Elastic Beanstalk, I got MemoryErrors when trying to install dependencies. Tried with t3.small & t3.medium instances and getting same problem. When removing the chromadb dependency from requirements.txt, app is successfully deployed.

Versions

Chromadb==0.3.21

Relevant log output

2023/04/27 18:38:51.943548 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed with error exit status 2. Stderr:ERROR: Exception:
Traceback (most recent call last):
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/commands/install.py", line 339, in run
    requirement_set = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 215, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 288, in __init__
    super().__init__(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 532, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 214, in unpack_url
    file = get_http_url(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 94, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/network/download.py", line 146, in __call__
    for chunk in chunks:
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/cli/progress_bars.py", line 304, in _rich_progress_bar
    for chunk in iterable:
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 96, in read
    self._close()
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 76, in _close
    self.__callback(result)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/cachecontrol/controller.py", line 331, in cache_response
    self.serializer.dumps(request, response, body),
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/cachecontrol/serialize.py", line 70, in dumps
    return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 885, in pack
    self._pack(obj)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 864, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 970, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 864, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 970, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 823, in _pack
    return self._buffer.write(obj)
MemoryError

[shba007]

It's same for Render.com with Exit Code 139. I Think the main problem is chroma always tries to import the vectorizer which is huge. If we can use the vectorizer and db part separately then we may be able to make it more efficient. If we can come up with weaivate like architecture which has a module and db separately. Then we may be able to deploy it in cloud

[calderwoodra]

I also ran into this issue a few days ago (t2.micro). I think (?) a simple (no-code) solution would be for chroma to offer precompiled wheels similar to numpy.

[reedwi]

Any guidance on priority of this one? Would like to use Chroma for some use cases, but the install is so large it makes it impossible to use on Heroku, Lambda, and small servers which is ideally where I would want the retriever functionality to live.

[calderwoodra]

I found a workaround, it takes a few steps:

1. Create another EC2 instance similar to your smaller one but with more memory
2. Build the hnsw wheel inside that instance
3. Copy the wheel over to your smaller instance (I uploaded the wheel to my own git repo)
4. install the hnsw wheel on your smaller instance
5. install pytorch
6. install chromadb
7. shut down the larger instance (so you aren't billed)

Here are the commands to create the hnsw wheel (output to the dist/ folder):

git clone https://github.com/nmslib/hnswlib.git
cd hnswlib
virtualenv --python=$(which python3.9) env
source env/bin/activate
pip install --upgrade pip numpy pybind11 setuptools wheel
python setup.py bdist_wheel

To install hnsw:

pip install <full file name of wheel file with .whl extension>

Here is the command to install torch reliably:

pip install torch --index-url https://download.pytorch.org/whl/cpu --no-cache-dir

[jeffchuber]

We think the right way to solve this is to reduce the install memory requirements by shipping pre-built HNSW binaries instead of requiring it to be built an install time.

[jeffchuber]

This is likely fixed with Chroma 0.4 - need to test however

[jeffchuber]

Closing this issue for now. Chroma 0.4+ is much smaller and easier to build.