exo Wishlist: Asahi Linux support

Linux has much more robust tools for cluster management than MacOS, and Asahi Linux kernels have support for all the relevant Apple M chip hardware. The goal is to make running a cluster with a proper OS level management beyond "run exo in tmux" possible on ARM Macs.

Ideal use cases:

Kubernetes with GPU passthrough
Docker Swarm with a /dev parameter
NFS Root shared OS clusters

Dec 27 '24 16:12 tidux

This sounds very interesting and also like an almost completely different project? What parts of the existing codebase would your proposal share?

Dec 28 '24 19:12 AFDudley

All it needs is M-chip support for the AI backend(s?) under Linux in the exo tool. The rest I could put together at the sysadmin level - Dockerfiles, Kubernetes manifests, etc. p2p cluster app management inside Kubernetes or Docker Swarm is a well known pattern.

Dec 28 '24 19:12 tidux

Can you confirm mlx works on an Asahi Linux based distro?

Dec 28 '24 22:12 AFDudley

It looks like mlx is Metal/MacOS only for now so this remains on hold until that changes.

Jan 04 '25 23:01 tidux

I think the this could work (the exo part, k8s is separate I guess) with the tinygrad OpenCL back-end and rusticl/mesa? I don't have a machine to try it unfortunately.

Jan 08 '25 16:01 pbouda

All it needs is M-chip support for the AI backend(s?) under Linux in the exo tool.

For that, you would need an MLX replacement that runs on Vulkan 1.3.

GPT4ALL may have such a component: https://www.nomic.ai/blog/posts/gpt4all-gpu-inference-with-vulkan

Jan 29 '25 03:01 silibattlebot