exo icon indicating copy to clipboard operation
exo copied to clipboard
Published 18 hours ago • verified publisher icon exo-explore

we need option to use intel graphics like (intel iris xe)

Open Kbassem10 opened this issue 1 year ago • 11 comments

we need option to use intel graphics like (intel iris xe) on wsl

Kbassem10 avatar Sep 28 '24 10:09 Kbassem10

@Kbassem10 Did some looking.

  1. It would need to be implement ultimately, someone would need to reference: https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/ollama_quickstart.md
  2. Many users are saying that the CPU code names can give them pass thru functionality. I.e. Coffeelake, Cooperlake, etc.

Could you share some more about your setup. I think some more research and this could be possible.

I'm also assuming you mean WSL2 when you state WSL?

larson-carter avatar Sep 30 '24 17:09 larson-carter

Definitely agree. There are a lot of mini pcs for this use case. So iGPU would be very welcome. I have three Minisforum MS-01 with Intel Iris Xe iGPUs which I would like to test with this.

I use plain linux by the way. If I try I get the error: Linux Box (🔴evice: CLANG) 62GB │ │ 0TFLOPS │ │ [0.00-1.00]

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): [Errno 2] No such file or directory: 'clang'

Thank you in advance.

djakupovic avatar Oct 01 '24 18:10 djakupovic

@djakupovic

Definitely agree. There are a lot of mini pcs for this use case. So iGPU would be very welcome. I have three Minisforum MS-01 with Intel Iris Xe iGPUs which I would like to test with this.

  • Great, this would be perfect for testing as we progress here.

I use plain linux by the way. If I try I get the error:

  • Great we can start with linux and make our way over to WSL once we get this working in plain linux.

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): [Errno 2] No such file or directory: 'clang'

  • Hmm, this is a new one for me(i only deploy on macOS), however I think that this might be a python package or deps issue.

Have you tried the off chance installing clang manually? Using <your_package_manager> install clang

larson-carter avatar Oct 01 '24 19:10 larson-carter

@djakupovic

Definitely agree. There are a lot of mini pcs for this use case. So iGPU would be very welcome. I have three Minisforum MS-01 with Intel Iris Xe iGPUs which I would like to test with this.

  • Great, this would be perfect for testing as we progress here.

I use plain linux by the way. If I try I get the error:

  • Great we can start with linux and make our way over to WSL once we get this working in plain linux.

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): [Errno 2] No such file or directory: 'clang'

  • Hmm, this is a new one for me(i only deploy on macOS), however I think that this might be a python package or deps issue.

Have you tried the off chance installing clang manually? Using <your_package_manager> install clang

@larson-carter I installed clang but still the same error

djakupovic avatar Oct 01 '24 19:10 djakupovic

@djakupovic

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): [Errno 2] No such file or directory: 'clang'

Just so were on the same page(and I can try to mimic this):

  • Are you getting this error when entering a prompt in the web UI?

  • What is your setup like? How many devices?

larson-carter avatar Oct 01 '24 19:10 larson-carter

@djakupovic

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): [Errno 2] No such file or directory: 'clang'

Just so were on the same page(and I can try to mimic this):

  • Are you getting this error when entering a prompt in the web UI?
  • What is your setup like? How many devices?

@larson-carter Yes after I entered a prompt. I tested with clang-14 default in Debian bookwork and even updated manually to clang-18

Setup currently just one device to test, afterwards I would try it with three.

You were right, clang was missing, apt install clang solved it. However, now I get this: Linux Box (🔴evice: CLANG) 62GB │ │ 0TFLOPS │ │ [0.00-1.00] │ │ │ │ │ loaded weights in 42.42 ms, 0.03 GB loaded at 0.79 GB/s ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ Traceback (most recent call last): ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/api/chatgpt_api.py", line 252, in handle_post_chat_completions await self.node.process_prompt(shard, prompt, image_str, request_id=request_id) ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/orchestration/standard_node.py", line 98, in process_prompt resp = await self._process_prompt(base_shard, prompt, image_str, request_id, inference_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/orchestration/standard_node.py", line 134, in _process_prompt result, inference_state, is_finished = await self.inference_engine.infer_prompt(request_id, shard, prompt, image_str, inference_state=inference_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/inference/tinygrad/inference.py", line 62, in infer_prompt await self.ensure_shard(shard) ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/inference/tinygrad/inference.py", line 100, in ensure_shard self.model = await asyncio.get_event_loop().run_in_executor(self.executor, build_transformer, model_path, shard, "8B" if "8b" in shard.model_id.lower() else "70B") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/exo/exo/inference/tinygrad/inference.py", line 51, in build_transformer load_state_dict(model, weights, strict=False, consume=False) # consume=True ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/nn/state.py", line 129, in load_state_dict else: v.replace(state_dict[k].to(v.device)).realize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/tensor.py", line 3500, in _wrapper ret = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/tensor.py", line 213, in realize run_schedule(*self.schedule_with_vars(*lst), do_update_stats=do_update_stats) ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 222, in run_schedule for ei in lower_schedule(schedule): ^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 215, in lower_schedule raise e ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 209, in lower_schedule try: yield lower_schedule_item(si) ^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 193, in lower_schedule_item runner = get_runner(si.outputs[0].device, si.ast) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 162, in get_runner method_cache[ckey] = method_cache[bkey] = ret = CompiledRunner(replace(prg, dname=dname)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/engine/realize.py", line 84, in init self.lib:bytes = precompiled if precompiled is not None else Device[p.dname].compiler.compile_cached(p.src) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/device.py", line 183, in compile_cached lib = self.compile(src) ^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/site-packages/tinygrad/runtime/ops_clang.py", line 15, in compile subprocess.check_output(['clang', '-shared', *self.args, '-O2', '-Wall', '-Werror', '-x', 'c', '-fPIC', '-ffreestanding', '-nostdlib', ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ File "/home/pmox/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, ╭──────────────────────────────────────────────────────────── Exo Cluster (1 node) ─────────────────────────────────────────────────────────────╮ subprocess.CalledProcessError: Command '['clang', '-shared', '-march=native', '-O2', '-Wall', '-Werror', '-x', 'c', '-fPIC', '-ffreestanding', '-nostdlib', '-', '-o', '/tmp/tmpd_oodaaj']' returned non-zero exit status 1.

djakupovic avatar Oct 01 '24 19:10 djakupovic

@djakupovic

Seems like we are getting somewhere.

This is giving us a slight hint:

subprocess.CalledProcessError: Command '['clang', '-shared', '-march=native', '-O2', '-Wall', '-Werror', '-x', 'c', '-fPIC', '-ffreestanding', '-nostdlib', '-', '-o', '/tmp/tmpd_oodaaj']' returned non-zero exit status 1.

Help me debug some more if you don't mind.

  1. Have you installed all of the deps/libs required by exo itself? If so how did you do that?
  2. Verify clang version: clang --version
  3. Freestanding and nostdlib have lots of deps lets try to install them all (if already installed it wont harm anything) sudo apt install build-essential
  4. Optional - echo "" | clang -shared -march=native -O2 -Wall -Werror -x c -fPIC -ffreestanding -nostdlib - -o /tmp/tmpd_oodaaj

I'm not as experienced with Debian usually I use Ubuntu and RHEL (macOS everyday) but I hope to be able to help you get this running then go from there.

larson-carter avatar Oct 01 '24 20:10 larson-carter

@djakupovic

Seems like we are getting somewhere.

This is giving us a slight hint:

subprocess.CalledProcessError: Command '['clang', '-shared', '-march=native', '-O2', '-Wall', '-Werror', '-x', 'c', '-fPIC', '-ffreestanding', '-nostdlib', '-', '-o', '/tmp/tmpd_oodaaj']' returned non-zero exit status 1.

Help me debug some more if you don't mind.

  1. Have you installed all of the deps/libs required by exo itself? If so how did you do that?
  2. Verify clang version: clang --version
  3. Freestanding and nostdlib have lots of deps lets try to install them all (if already installed it wont harm anything) sudo apt install build-essential
  4. Optional - echo "" | clang -shared -march=native -O2 -Wall -Werror -x c -fPIC -ffreestanding -nostdlib - -o /tmp/tmpd_oodaaj

I'm not as experienced with Debian usually I use Ubuntu and RHEL (macOS everyday) but I hope to be able to help you get this running then go from there.

@larson-carter

  1. Yes by git clone https://github.com/exo-explore/exo.git cd exo pip install .
  2. Debian clang version 18.1.8 (++20240731024826+3b5b5c1ec4a3-1~exp1~20240731144843.145) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

With 14 the same error

  1. sudo apt install build-essential --> already installed
  2. run the command and tried again, still the same error: Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): Command '['clang', '-shared', '-march=native', '-O2', '-Wall', '-Werror', '-x', 'c', '-fPIC', '-ffreestanding', '-nostdlib', '-', '-o', '/tmp/tmpgce0ug4f']' returned non-zero exit status 1.

djakupovic avatar Oct 01 '24 20:10 djakupovic

See #458 for a fix to your CLANG build error.

Came here to see if Intel GPUs were getting any love in here.

DrEVILish avatar Dec 04 '24 23:12 DrEVILish

@DrEVILish were you able to get anywhere further with Intel GPUs?

I've got an i9-14900k, 64GB DDR5 and Battlemage B580 setup that I'm successfully running ollama per the ipex-llm documentation: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/bmg_quickstart.md

There is also instructions provided for pytorch and llama.cpp.

Setting up a new conda environment, installing pytorch (following ipex-llm's instructions), and then installing exo with pip, I'm close? but it bails out, trying to load nvidia libraries.

Same thing with llama.cpp. New conda environment, ipex-llm first, then exo. Tries to load nvidia libraries.

Anyway, just wanted to see if others have made further progress utilizing intel hardware here. I've attached my sessions documenting my attempts if that helps the discussion/others.

2025-02-20_0001_ipex-llm-pytorch_exo-main.txt 2025-02-20_0001_ipex-llm-llama_exo-main.txt

shellfyred avatar Feb 20 '25 18:02 shellfyred

I didn't bother trying again, seemed to be lacking support from devs so far. Wasn't worth the time nor a priority for me just playing about. Sorry

DrEVILish avatar Feb 20 '25 18:02 DrEVILish