Build latest triton custom image for Ubuntu 18.04

[gandharv-kapoor]

Is your feature request related to a problem? Please describe. On Azure VM's the Ubuntu OS being used is still 18.04 and therefore we are forced to use and older triton image (<= 20.11) therefore missing out on all the latest optimizations and features. We therefore would like to create our own custom image with latest triton release on Ubuntu 18.04. We were also hoping to use only specific backends (Onnx, Python & PyTorch) with this image to lower down the image size as much as possible.

Before trying this out wanted to check regarding the feasibility of this solution and whether it's a best solution for our going forward.

I was also hoping to get some suggestion on using compose.py vs build.py for achieving this. If i use compose.py can I specify min image to be 20.11 and max to be latest 22.06 to get all the latest features on a 20.11 image and build this on a Ubuntu 18.04 system or can I just directly use compose.py and use latest 22.06 on a Ubuntu 18.04 system and that should work as well? The other option is to use build.py which I haven't yet explored in detail.

[rmccorm4]

Hi @gandharv-kapoor ,

Are you trying to install Triton directly on the VM, or are you trying to run Triton containers (docker) on the VM?

If you are building/running docker images, then it shouldn't matter that the VM's Ubuntu version is 18.04 - you can run a 20.04 ubuntu docker container on an older ubuntu host with docker installed. ex: running docker run -ti ubuntu:20.04 on your VM should work with docker installed (as well as latest nvcr.io/nvidia/tritonserver images).

[gandharv-kapoor]

Hi @rmccorm4

Thanks for your response and getting to this ticket. We actually were doing that originally the later (run a 20.04 ubuntu docker container on an older ubuntu host with docker installed with latest tritonserver) but it lead to deadlock and we saw in the backtrace that the code was stuck in pthread library and thats why we had to move to older triton version which works directly with 18.04 but we are are missing out on a lot newer features and optimizations. Hopefully this gives you the context on our motivation for being able to build the custom image of latest triton on ubuntu 18.04.

[gandharv-kapoor]

Hi @rmccorm4 @dyastremsky @Tabrizian

Any suggestion to my previous question? We are not able to use latest triton images on older Ubuntu 18 VMs therefore any help on what might the best workaround, much appreciated.

[Tabrizian]

Hi @gandharv-kapoor I think your best bet would be to prepare your own base container that has all the dependencies that the tritonserver min image has and use 18.04 ubuntu instead of 20.04. There are some cuda containers that might help you with some of the dependencies but I think you still need to install some additional dependencies too.

After you have prepared the base container, you can use the build.py script to build triton based on the base image that you have created. The compose.py script can only be used for customizing the container created by build.py and doesn't build backends/server from source.

[gandharv-kapoor]

Hi @gandharv-kapoor I think your best bet would be to prepare your own base container that has all the dependencies that the tritonserver min image has and use 18.04 ubuntu instead of 20.04. There are some cuda containers that might help you with some of the dependencies but I think you still need to install some additional dependencies too.

After you have prepared the base container, you can use the build.py script to build triton based on the base image that you have created. The compose.py script can only be used for customizing the container created by build.py and doesn't build backends/server from source.

@Tabrizian are you suggesting to cherry pick dependencies to create my own container? Cant I use an existing docker file with dependencies listed for triton? Is there an easier work around to create this docker file of dependencies.

Also wont just building a custom image using compose.py on ubuntu 18.04 work? I wonder why that is not a possible workaround?

[Tabrizian]

Cant I use an existing docker file with dependencies listed for triton?

The problem is that the Dockerfile for the min container is not published. For building Triton on Ubuntu 18.04, you would need to have a specific CUDA version. Because of this I think you would need to build your own min container image.

Also wont just building a custom image using compose.py on ubuntu 18.04 work?

Compose.py can be only used for customizing an already built triton image. It is definitely possible to use that for further customizing the image after you have built it. But it is not useful for building Triton from source.

[gandharv-kapoor]

Cant I use an existing docker file with dependencies listed for triton?

The problem is that the Dockerfile for the min container is not published. For building Triton on Ubuntu 18.04, you would need to have a specific CUDA version. Because of this I think you would need to build your own min container image.

@Tabrizian any particular reason why is it not published or available? Can it be something provided by the triton team? It would be really nice to have this even for others who might want to customize things or fall into same problem working with older ubuntu.

If not I am not 100% sure if this will help creating the dockerfile https://stackoverflow.com/questions/19104847/how-to-generate-a-dockerfile-from-an-image

Also would be helpful if you could provide more clarity and details on how go about "building your own min triton container image." what would be a good starting point and how to go about it as I would be doing this for the first time.

Thanks appreciate all the advice @Tabrizian

Also wont just building a custom image using compose.py on ubuntu 18.04 work?

Compose.py can be only used for customizing an already built triton image. It is definitely possible to use that for further customizing the image after you have built it. But it is not useful for building Triton from source.

[jbkyang-nvi]

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

[mikelam92]

Hi @Tabrizian we might go down the path of creating the image ourselves. And we would like to follow up on the questions here. Thank you!

[jbkyang-nvi]

Cant I use an existing docker file with dependencies listed for triton?

The problem is that the Dockerfile for the min container is not published. For building Triton on Ubuntu 18.04, you would need to have a specific CUDA version. Because of this I think you would need to build your own min container image.

@Tabrizian any particular reason why is it not published or available? Can it be something provided by the triton team? It would be really nice to have this even for others who might want to customize things or fall into same problem working with older ubuntu.

We do have a min container published, just not one for Ubuntu 18.04. You will need to look at the dependencies that exist for our current min container (eg: nvcr.io/nvidia/tritonserver:22.11-py3-min) and replicate that. We currently don't have enough demands for older container support to justify more support for older Ubuntu containers.

If not I am not 100% sure if this will help creating the Dockerfile https://stackoverflow.com/questions/19104847/how-to-generate-a-dockerfile-from-an-image

It looks helpful to creating the Dockerfile. The min image Triton provides has custom Nvidia binaries used, so the generated Dockerfile will probably not directly usable.

Also would be helpful if you could provide more clarity and details on how go about "building your own min triton container image." what would be a good starting point and how to go about it as I would be doing this for the first time.

Thanks appreciate all the advice @Tabrizian

I would suggest you start by using the the min CUDA containers as a base container and build your dockerfile from there as Iman suggested. Please refer to the Triton Support Matrix for version suggestions (11.7 for Triton 22.06).

[jbkyang-nvi]

Also, there's an official support path for Nvidia products that could potentially help you with this issue. https://www.nvidia.com/en-us/data-center/products/ai-enterprise/#benefits

[krishung5]

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue.