exo No models downloading

Since the implementation of Llava the downloads doesn't start.

Jul 31 '24 12:07 ManuXD32

Probably duplicate of https://github.com/exo-explore/exo/issues/107 Fixed with https://github.com/exo-explore/exo/commit/76766253cde775f58f5637344dae2656f1c1447a

Jul 31 '24 21:07 AlexCheema

Probably duplicate of #107 Fixed with 7676625

It keeps happening, there isn't even the log that anaunces the connection like "Connected from ******!

Aug 01 '24 06:08 ManuXD32

It's the same for me.

Aug 01 '24 08:08 govertbuijs

Reopening. Can you give some more info - what commit are you running on? What device? Do you get any logs? Can you try running with DEBUG=2 e.g. DEBUG=2 python3 main.py

Aug 01 '24 08:08 AlexCheema

commit f5755ea198fe2e7601624bd2df8fe79ace5f1b4b

Handling chat completions request from 127.0.0.1: {'model': 'llama-3-8b', 'messages': [{'role': 'user', 'content': 'why is the sky blue'}], 'stream': True}
Trying AutoProcessor for llama3-8b-sfr
Failed to load processor for llama3-8b-sfr. Error: llama3-8b-sfr is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`

Interestingly enough, on another box Mistral Nemo started downloading.

Aug 01 '24 08:08 govertbuijs

Reopening. Can you give some more info - what commit are you running on? What device? Do you get any logs? Can you try running with DEBUG=2 e.g. DEBUG=2 python3 main.py

I'm running the latest commit on main branch. Using a HP Pavilion Laptop 15-eh1xxx. CPU: AMD Ryzen 5 5500U (12) @ 4,06 GHz GPU: AMD Lucienne [Integrated] Memory: 30,67 GiB

DEBUG=2 CLANG=1 python main.py

/ _ \ / / _ \ | /> < (_) | _/_/____/

Detected system: Linux Using inference engine: TinygradDynamicShardInferenceEngine Trying to find available port port=62858 [61135, 62399, 63541, 60410, 51418, 60240, 59972, 56332, 58608, 56169, 63996, 61066, 55950, 50458, 61500, 54886, 63400, 53337, 61271] Using available port: 62858 Retrieved existing node ID: 245129e2-d7c4-4bd9-a170-66ed640225a8 tinygrad Device.DEFAULT='CLANG' Server started, listening on 0.0.0.0:62858 tinygrad Device.DEFAULT='CLANG' Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Collected topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Chat interface started. Open this link in your browser: http://localhost:8000 ChatGPT API endpoint served at http://localhost:8000/v1/chat/completions Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Received request: GET /index.js Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Received request: POST /v1/chat/completions Handling chat completions request from 127.0.0.1: {'model': 'llama-3.1-8b', 'messages': [{'role': 'user', 'content': 'Hello'}], 'stream': True} Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {}) Starting with the following peers: [] Connecting to new peers... Collecting topology max_depth=4 visited=set() Topology collection task executed. Current topology: Topology(Nodes: {245129e2-d7c4-4bd9-a170-66ed640225a8: Model: Linux Box (Device: CLANG). Chip: Unknown Chip (Device: CLANG). Memory: 31405MB. Flops: fp32: 0.00 TFLOPS, fp16: 0.00 TFLOPS, int8: 0.00 TFLOPS}, Edges: {})

Aug 01 '24 09:08 ManuXD32