exo
exo copied to clipboard
Cut down exo package size
- We are going to package exo as an installable (first step is nix package see: https://github.com/exo-explore/exo/pull/210)
- We want the exo installable to be lightweight with minimal dependencies
Dependencies we can remove
- [ ] Transformers #169
- [ ] hf-transfer #264
- [ ] huggingface-hub #264
- [ ] Jinja2 #264
- [ ] lxml
- [ ] sentencepiece
- [ ] tiktoken
Full list of dependencies and their size
+---------------+-------------+
| Package | Size (MB) |
+===============+=============+
| transformers | 81.49 |
+---------------+-------------+
| mlx | 81.41 |
+---------------+-------------+
| numpy | 28.05 |
+---------------+-------------+
| lxml | 20.03 |
+---------------+-------------+
| tinygrad | 12.53 |
+---------------+-------------+
| pip | 10.32 |
+---------------+-------------+
| setuptools | 7.92 |
+---------------+-------------+
| pygments | 7.77 |
+---------------+-------------+
| tokenizers | 7.01 |
+---------------+-------------+
| sentencepiece | 2.9 |
+---------------+-------------+
| tiktoken | 2.65 |
+---------------+-------------+
| aiohttp | 2.25 |
+---------------+-------------+
| psutil | 2.06 |
+---------------+-------------+
| rich | 1.98 |
+---------------+-------------+
| regex | 1.65 |
+---------------+-------------+
| fsspec | 1.4 |
+---------------+-------------+
| jinja2 | 1.1 |
+---------------+-------------+
| safetensors | 1.02 |
+---------------+-------------+
| urllib3 | 0.8 |
+---------------+-------------+
| idna | 0.59 |
+---------------+-------------+
| mashumaro | 0.58 |
+---------------+-------------+
| blobfile | 0.58 |
+---------------+-------------+
| orjson | 0.52 |
+---------------+-------------+
| wheel | 0.42 |
+---------------+-------------+
| yarl | 0.38 |
+---------------+-------------+
| requests | 0.38 |
+---------------+-------------+
| tqdm | 0.35 |
+---------------+-------------+
| pynvml | 0.34 |
+---------------+-------------+
| packaging | 0.32 |
+---------------+-------------+
| certifi | 0.29 |
+---------------+-------------+
| typeguard | 0.26 |
+---------------+-------------+
| inflect | 0.21 |
+---------------+-------------+
| tabulate | 0.18 |
+---------------+-------------+
| tenacity | 0.16 |
+---------------+-------------+
| frozenlist | 0.15 |
+---------------+-------------+
| MarkupSafe | 0.15 |
+---------------+-------------+
| platformdirs | 0.14 |
+---------------+-------------+
| multidict | 0.14 |
+---------------+-------------+