[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).

[Rohan-Jalil]

Command:

llama stack run Llama3.2-11B-Vision-Instruct --port 5000

Output:

Using config `/Users/mac/.llama/builds/conda/Llama3.2-11B-Vision-Instruct-run.yaml`
Resolved 4 providers
 inner-inference => meta-reference
 models => __routing_table__
 inference => __autorouted__
 inspect => __builtin__

[2024-10-15 07:20:46,247] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.
Loading model `Llama3.2-11B-Vision-Instruct`
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 50714) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).

config `/Users/mac/.llama/builds/conda/Llama3.2-11B-Vision-Instruct-run.yaml

version: '2'
built_at: '2024-10-15T06:25:20.837875'
image_name: Llama3.2-11B-Vision-Instruct
docker_image: null
conda_env: Llama3.2-11B-Vision-Instruct
logging:
  level: DEBUG
  file:
    enabled: true
    path: /Users/mac/llama-32.log
authentication:
  enabled: false
apis:
- inference
providers:
  inference:
  - provider_id: meta-reference
    provider_type: meta-reference
    config:
      model: Llama3.2-11B-Vision-Instruct
      torch_seed: null
      max_seq_len: 4096
      max_batch_size: 1

Command:

python -m llama_stack.apis.inference.client localhost 5000 --no-auth

Output

User>hello world, write me a 2 sentence poem about the moon

Error: HTTP 403

[Itime-ren]

Your issue is similar to mine： https://github.com/meta-llama/llama-stack/issues/194

[yanxi0830]

Could you try with adding the --disable-ipv6 flag?

llama stack run Llama3.2-11B-Vision-Instruct --port 5000 --disable-ipv6

[Rohan-Jalil]

@yanxi0830 Output of the command llama stack run Llama3.2-11B-Vision-Instruct --port 5000 --disable-ipv6 is mentioned below:

Resolved 4 providers inner-inference => meta-reference models => routing_table inference => autorouted inspect => builtin

[2024-10-16 14:23:29,360] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. Loading model Llama3.2-11B-Vision-Instruct [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known). [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known). [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known). [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known). [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known). [W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 58737) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).

[pySilver]

Same thing with docker setup

[pySilver]

curl -v http://127.0.0.1:5000/inference/chat_completion \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Llama3.1-1B-Instruct",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write me a 2 sentence poem about the moon"}
    ],
    "sampling_params": {"temperature": 0.7, "seed": 42, "max_tokens": 512}
}'
*   Trying 127.0.0.1:5000...
* Connected to 127.0.0.1 (127.0.0.1) port 5000
> POST /inference/chat_completion HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 292
>
* upload completely sent off: 292 bytes
< HTTP/1.1 403 Forbidden
< Content-Length: 0
< Server: AirTunes/775.3.1
<
* Connection #0 to host 127.0.0.1 left intact

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant!