tensorrt

tensorrt copied to clipboard

Converting BERT to TensorRT FP16 model added a lot of inputs named "unknown"

5

Versions: - Tensorflow 2.2 (TensorRT 6, CUDA 10) After pulling a transformer model, specifically `albert-base-v2` from Huggingface: ```python albert = TFAlbertModel.from_pretrained('albert-base-v2') ``` I wanted to run TensorRT inference on the...

statikkkkk

Run Bert-like model failed in tensorflow-trt container: Could not initialize cublas

1

I have a model like Bert running on tensorflow 1. X, and the hardware is T4. And I'm trying to speed it up with TRT. I found that NGC has...

my777777

I can not use conversion on Jetson Nano with [this guide](https://github.com/tensorflow/tensorrt/tree/master/tftrt/triton). The problem is that TensorFlow container doesn't start: ``` docker: Error response from daemon: failed to create shim: OCI...

ArgoHA

## System information ### NVIDIA ```bash $ nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC...

sayakpaul

Is it possible to use tensorrt to speed up original tensorflow t5 exported saved_model?

4

i've tried huggingface t5 model speed up by trt, but how can we speed up tensorflow t5 saved_model? i want to use speed-up t5 saved_model in tf-serving for production env....

chenl-chief

Hi, i am trying to convert Transformer model with tensorRT ops, the model is trained based on Opennmt-tf. here is my scripts: CUDA_VISIBLE_DEVICES=0 saved_model_cli convert --dir ./averaged-ende-export500k-v2 --tag_set serve --output_dir...

lyzKF

Why no improvemenmt in the object_detection example

12

tensorflow-gpu1.14 CUDA10.1 GPU gtx1080 my_test.json { "model_config": { "model_name": "ssd_resnet_50_fpn_coco", "input_dir": "/home/liujt/software/tensorrt/data", "batch_size": 8, "override_nms_score_threshold": 0.3 }, "optimization_config": { "use_trt": true, "precision_mode": "INT8", "calib_images_dir": "/home/liujt/software/tensorrt/data/train2017", "num_calib_images": 8, "calib_batch_size": 8, "calib_image_shape":...

chinesesoft8

A converted model contains the frozen variables (const), the original variables, and the variables saved into the TRT engine as weights. This can lead up to 3x size of the...

tfeher

Loading the file to build the model failed

2

When I load the model using the tensorRT model file generated by TF-TRT, the console displays the following information: 2022-07-13 16:42:54.914735: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger coreReadArchive.cpp (38) - Serialization Error in...

AarenWu

Cannot convert TF-Text Tokenizer to TensorRT

4

**System information** - Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No - OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu...

dshahrokhian