tensorrtllm_backend icon indicating copy to clipboard operation
tensorrtllm_backend copied to clipboard

[Bugfix] Launch Triton server without waiting for a signal

Open michaelnny opened this issue 1 year ago • 0 comments

Hi,

Problem: This PR fix a silent bug inside the scripts\launch_triton_server.py module, this issue only occurs if we try to automatically launch the triton server inside a container using either CMD in the Dockerfile, or command in the docker-compose.yaml file.

For example in a Dockerfile:

CMD ["python3", "scripts\launch_triton_server.py", "--model_repo", "/workspace/model_repos/llama3_ifb", "--world_size", "1"]

Cause: The cause of the issue is we're not wait for the signal when we call subprocess.Popen(cmd, env=env)

Before the change:

    subprocess.Popen(cmd, env=env) 

After the change:


    # Start the subprocess and wait for signal
    with subprocess.Popen(cmd, env=env) as proc:
        try:
            retcode = proc.wait()
        except KeyboardInterrupt:
            proc.kill()
            return 0

michaelnny avatar May 23 '24 09:05 michaelnny