worker-vllm icon indicating copy to clipboard operation
worker-vllm copied to clipboard

Adds function to parse JSON env var

Open FuxMak opened this issue 6 months ago • 0 comments

Resolves #192 and #195.

I'm not an expert in Python, but the function was required for me to launch a vllm-worker with a Qwen3-30B-A3B and more than the 32k context window, as described in the runpod.io blogpost.

With the following environment variable and the changes, the container was able to run and pass the settings successfully to vllm. The function parse_json_env_var in my PR is not only used in the ROPE_SCALING var, but already all vars of type dict, as they will face the same issue.

ROPE_SCALING: {"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}

FuxMak avatar Jul 07 '25 05:07 FuxMak