exo icon indicating copy to clipboard operation
exo copied to clipboard

Cut down exo package size

Open AlexCheema opened this issue 1 year ago • 0 comments

  • We are going to package exo as an installable (first step is nix package see: https://github.com/exo-explore/exo/pull/210)
  • We want the exo installable to be lightweight with minimal dependencies

Dependencies we can remove

  • [ ] Transformers #169
  • [ ] hf-transfer #264
  • [ ] huggingface-hub #264
  • [ ] Jinja2 #264
  • [ ] lxml
  • [ ] sentencepiece
  • [ ] tiktoken

Full list of dependencies and their size

+---------------+-------------+
| Package       |   Size (MB) |
+===============+=============+
| transformers  |       81.49 |
+---------------+-------------+
| mlx           |       81.41 |
+---------------+-------------+
| numpy         |       28.05 |
+---------------+-------------+
| lxml          |       20.03 |
+---------------+-------------+
| tinygrad      |       12.53 |
+---------------+-------------+
| pip           |       10.32 |
+---------------+-------------+
| setuptools    |        7.92 |
+---------------+-------------+
| pygments      |        7.77 |
+---------------+-------------+
| tokenizers    |        7.01 |
+---------------+-------------+
| sentencepiece |        2.9  |
+---------------+-------------+
| tiktoken      |        2.65 |
+---------------+-------------+
| aiohttp       |        2.25 |
+---------------+-------------+
| psutil        |        2.06 |
+---------------+-------------+
| rich          |        1.98 |
+---------------+-------------+
| regex         |        1.65 |
+---------------+-------------+
| fsspec        |        1.4  |
+---------------+-------------+
| jinja2        |        1.1  |
+---------------+-------------+
| safetensors   |        1.02 |
+---------------+-------------+
| urllib3       |        0.8  |
+---------------+-------------+
| idna          |        0.59 |
+---------------+-------------+
| mashumaro     |        0.58 |
+---------------+-------------+
| blobfile      |        0.58 |
+---------------+-------------+
| orjson        |        0.52 |
+---------------+-------------+
| wheel         |        0.42 |
+---------------+-------------+
| yarl          |        0.38 |
+---------------+-------------+
| requests      |        0.38 |
+---------------+-------------+
| tqdm          |        0.35 |
+---------------+-------------+
| pynvml        |        0.34 |
+---------------+-------------+
| packaging     |        0.32 |
+---------------+-------------+
| certifi       |        0.29 |
+---------------+-------------+
| typeguard     |        0.26 |
+---------------+-------------+
| inflect       |        0.21 |
+---------------+-------------+
| tabulate      |        0.18 |
+---------------+-------------+
| tenacity      |        0.16 |
+---------------+-------------+
| frozenlist    |        0.15 |
+---------------+-------------+
| MarkupSafe    |        0.15 |
+---------------+-------------+
| platformdirs  |        0.14 |
+---------------+-------------+
| multidict     |        0.14 |
+---------------+-------------+

AlexCheema avatar Oct 02 '24 13:10 AlexCheema