RuntimeError - "Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1"

[owen198]

Configuration:

1. Mac M2 max: 64GB RAM
2. Linux NVIDIA 4070ti

Description: I can start and connect devices and access tinychat from two different devices (Mac M2 max 64GB and Linux with 4070ti). However, after submitting a message, I encountered the following error:

Error processing prompt (see logs with DEBUG>=2): <AioRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Unexpected <class 'RuntimeError'>: Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1."
    debug_error_string = "UNKNOWN:Error received from peer {created_time:\"2024-10-21T20:22:22.048140119+08:00\", grpc_status:2, grpc_message:\"Unexpected <class 'RuntimeError'>: Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1.\"}"
>

Environment Details:

• Model: LLaMA 3.1, 8B
• Python Version: 3.12
• CUDA Version: 12.2
• Nvidia Driver: 535.183.06

Observations: When I shut down the Linux box, everything works fine, and the LLaMA model responds accordingly on the Mac. The error only occurs when the Linux box joins the cluster, which suggests an issue when both devices are connected.

[UmutAlihan]

having the same issue:

{"detail": "Error processing prompt (see logs with DEBUG>=2): Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1."}%

[RoryMB]

Same issue.

Machines:

• M1 Max MacBook Pro (64GB RAM)
• M4 Max MacBook Pro (128GB RAM)

Model: models--mlx-community--Llama-3.1-Nemotron-70B-Instruct-HF-bf16

Error:

Error: Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Unexpected <class 'RuntimeError'>: Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1."
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Unexpected <class \'RuntimeError\'>: Item size 2 for PEP 3118 buffer format string B does not match the dtype B item size 1.", grpc_status:2, created_time:"2024-11-27T10:20:48.678437-06:00"}"

[dakecrazy]

no one know what and know how？

[dakecrazy]

@AlexCheema boss，can you solve this issue

[rltakashige]

Should no longer be an issue in 1.0 or future planned updates - but this happens as numpy does not support bfloat16 as a dtype.