TensorRT
TensorRT copied to clipboard
pytorch
PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
We currently have multiple compile functions in the file `py/torch_tensorrt/dynamo/_compiler.py`, which makes bad maintainability. The reason why we previously separated these functions is for generating clear docstring. Thus, ideally, we...
The `ptq.h` file is gone in the new Torch-TensorRT versions. But the tests still use [it](https://github.com/pytorch/TensorRT/blob/c9b8bf58304b28c69956db36c6176d03036523ce/tests/accuracy/test_dla_int8_accuracy.cpp#L4).
# Problem Description Torch-TensorRT compilation for large models (such as LLMs and diffusion models) can consume excessive CPU and GPU memory. This often leads to freezes, CUDA OOM errors, TensorRT...
## TL;DR We want to minimize the resource consumption needed to build and run Torch-TRT Programs ## Goal(s) ## Tasks ```[tasklist] ### Tasks ``` ## Additional context
## Bug Description The `lower_example.py` Is Broken In `nvcr.io/nvidia/pytorch:25.10-py3`. ## To Reproduce 1. Run NVIDIA PyTorch Docker container using `docker run -it --rm --gpus all -v $(pwd):/mnt -w /mnt nvcr.io/nvidia/pytorch:25.10-py3`....
# Description Ensures that `torchtrtc` precision settings do not always contain a default `fp32` precision when the precision is explicitly passed as an argument. This is particularly important when compiling...
## Bug Description FAILED conversion/test_pdist_aten.py::TestDynamicShapePdistConverter::test_pdist_float_4_dim0_dynamic_dim1_dynamic_p_other - AssertionError: Tensor-likes are not close! Mismatched elements: 4 / 6 (66.7%) Greatest absolute difference: 0.291101336479187 at index (5,) (up to 0.005 allowed) Greatest relative...
## Bug Description After KV caching, the exported_program.module() fails with input not found error. Likely something changed in exported_program.module() API. Workaround is setting offload_module_to_cpu=False ## To Reproduce Steps to reproduce...
Bug: FAILED conversion/test_scalar_tensor_aten.py::TestScalarTensorConverter::test_scalar_tensor_float_1 FAILED conversion/test_index_aten.py::TestIndexConverter::test_index_zero_two_dim_ITensor_mask TRT 10.13.3.9 Pytorch 2.10.0a0+b558c986e8 Error: ``` 2025-10-11T19:58:31.844970Z 01O ------------------------------ Captured log call ------------------------------- 2025-10-11T19:58:31.844990Z 01O WARNING torch_tensorrt [TensorRT Conversion Context]:logging.py:24 Environment variable NVIDIA_TF32_OVERRIDE=0 but BuilderFlag::kTF32...
**Is your feature request related to a problem? Please describe.** When supporting the GROOT N1 model, we were not able to handle attn mask and had to use the position...