TensorRT
TensorRT copied to clipboard
NVIDIA
Could not find any implementation for node X.weight
Hello, I used MobilenetXT model for 8-bit quantization, but I reported this error when onnx was converted to TENSORRT. What is the reason? There was no error when using MobilenetV2
I used tensorrt 8.6.1.6
MobilenetXT The structure is as follows：
(start_block): Sequential( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) ) (SandGlass1): Sequential( (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False) (9): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass2): Sequential( (0): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=96, bias=False) (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(96, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(32, 144, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False) (9): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass3): Sequential( (0): Conv2d(144, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (4): ReLU6(inplace=True) (5): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False) (6): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass4): Sequential( (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (9): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass5): Sequential( (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (9): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass6): Sequential( (0): Conv2d(192, 48, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): Conv2d(48, 288, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (4): ReLU6(inplace=True) (5): Conv2d(288, 288, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=288, bias=False) (6): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass7): Sequential( (0): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=288, bias=False) (1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(288, 48, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(48, 288, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=288, bias=False) (9): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass8): Sequential( (0): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=288, bias=False) (1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(288, 48, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(48, 288, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=288, bias=False) (9): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass9): Sequential( (0): Conv2d(288, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (4): ReLU6(inplace=True) ) (SandGlass10): Sequential( (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (9): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass11): Sequential( (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (9): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass12): Sequential( (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (9): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass13): Sequential( (0): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (4): ReLU6(inplace=True) (5): Conv2d(576, 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False) (6): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass14): Sequential( (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (9): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass15): Sequential( (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (9): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass16): Sequential( (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (9): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass17): Sequential( (0): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (4): ReLU6(inplace=True) ) (SandGlass18): Sequential( (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (9): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (SandGlass19): Sequential( (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(960, 208, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): Conv2d(208, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): ReLU6(inplace=True) (8): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280, bias=False) (9): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (classifier): Sequential( (0): Dropout(p=0.2, inplace=False) (1): Linear(in_features=1280, out_features=6, bias=True) ) )
MobileNetV2 The structure is as follows： (start_block): Sequential( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) ) (InvertedResidual1): Sequential( (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual2): Sequential( (0): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False) (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual3): Sequential( (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False) (4): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(144, 24, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual4): Sequential( (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(144, 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False) (4): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(144, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual5): Sequential( (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (4): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual6): Sequential( (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False) (4): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual7): Sequential( (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False) (4): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual8): Sequential( (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (4): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual9): Sequential( (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (4): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual10): Sequential( (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (4): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual11): Sequential( (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False) (4): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual12): Sequential( (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (4): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual13): Sequential( (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False) (4): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual14): Sequential( (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(576, 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False) (4): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual15): Sequential( (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (4): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual16): Sequential( (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (4): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (InvertedResidual17): Sequential( (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) (3): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False) (4): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU6(inplace=True) (6): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (last_Conv2dNormActivation): Sequential( (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) ) (conv2): Conv2d(1280, 6, kernel_size=(1, 1), stride=(1, 1)) )
SandGlass4.5.weight + /SandGlass4/SandGlass4.5/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.5/Conv + PWN(/SandGlass4/SandGlass4.7/Clip) (CaskConvolution[0x80000009]) profiling completed in 0.336847 seconds. Fastest Tactic: 0x955d593b1135a423 Time: 0.00395457 [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.5.weight + /SandGlass4/SandGlass4.5/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.5/Conv + PWN(/SandGlass4/SandGlass4.7/Clip) (CaskFlattenConvolution[0x80000036]) [01/03/2024-09:58:05] [TRT] [V] CaskFlattenConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] >>>>>>>>>>>>>>> Chose Runner Type: CaskConvolution Tactic: 0x955d593b1135a423 [01/03/2024-09:58:05] [TRT] [V] =============== Computing costs for SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add [01/03/2024-09:58:05] [TRT] [V] *************** Autotuning format combination: Int8(768,16:4,4,1), Float(3072,16,4,1) -> Float(3072,16,4,1) *************** [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskConvolution[0x80000009]) [01/03/2024-09:58:05] [TRT] [V] CaskConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskFlattenConvolution[0x80000036]) [01/03/2024-09:58:05] [TRT] [V] CaskFlattenConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] *************** Autotuning format combination: Int8(96,16:32,4,1), Float(3072,16,4,1) -> Float(3072,16,4,1) *************** [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskConvolution[0x80000009]) [01/03/2024-09:58:05] [TRT] [V] CaskConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskFlattenConvolution[0x80000036]) [01/03/2024-09:58:05] [TRT] [V] CaskFlattenConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] *************** Autotuning format combination: Int8(96,16:32,4,1), Float(96,16:32,4,1) -> Float(96,16:32,4,1) *************** [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskConvolution[0x80000009]) [01/03/2024-09:58:05] [TRT] [V] CaskConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] --------------- Timing Runner: SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add (CaskFlattenConvolution[0x80000036]) [01/03/2024-09:58:05] [TRT] [V] CaskFlattenConvolution has no valid tactics for this config, skipping [01/03/2024-09:58:05] [TRT] [V] Deleting timing cache: 39 entries, served 0 hits since creation. [01/03/2024-09:58:05] [TRT] [E] 10: Could not find any implementation for node SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add. [01/03/2024-09:58:05] [TRT] [E] 10: [optimizer.cpp::nvinfer1::builder::cgraph::LeafCNode::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add.) Failed to create the engine
What is the reason for this error? Can we just stratify and skip him?
Looks like a bug, could you please provide the onnx and reproduce command?
onnx model is: mobilenet_qat.zip
deploy code is:
import os
import tensorrt as trt
from calibration import Calibrator
import pycuda.driver as cuda
import pycuda.autoinit
TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)
calib_images = r"XXX"
onnx_model_path = r"mobilenet_qat.onnx"
# create tensorrt-engine
def get_engine(onnx_file_path="", engine_file_path="", calibrator=None, save_engine=False):
with trt.Builder(TRT_LOGGER) as builder, \
builder.create_builder_config() as config, \
builder.create_network(1) as network, \
trt.Runtime(TRT_LOGGER) as runtime, \
trt.OnnxParser(network, TRT_LOGGER) as parser:
# parse onnx model file
if not os.path.exists(onnx_file_path):
quit('ONNX file {} not found'.format(onnx_file_path))
print('Loading ONNX file from path {}...'.format(onnx_file_path))
with open(onnx_file_path, 'rb') as model:
print('Beginning ONNX file parsing')
parser.parse(model.read())
assert network.num_layers > 0, 'Failed to parse ONNX model. \
Please check if the ONNX model is compatible '
print('Completed parsing of ONNX file')
print('Building an engine from file {}; this may take a while...'.format(onnx_file_path))
config.set_flag(trt.BuilderFlag.INT8)
config.int8_calibrator = calibrator
print('Int8 mode enabled')
plan = builder.build_serialized_network(network, config)
if plan is None:
print('Failed to create the engine')
return None
print("Completed creating the engine")
engine = runtime.deserialize_cuda_engine(plan)
if save_engine:
with open(engine_file_path, "wb") as f:
f.write(engine.serialize())
return engine
def run_int8_quantization():
print('*** onnx to tensorrt int8 engine ***')
calib = Calibrator(calib_images, (32, 32), "mycalib.cache")
engine_model_path = "SP_mobilenet_int8.engine"
runtime_engine = get_engine(onnx_model_path, engine_model_path, calib, save_engine=True)
assert runtime_engine, 'failed engine generation...'
print('*** success to generate INT8 engine file ***\n')
if __name__ == '__main__':
run_int8_quantization()
Is this the error you see?
[01/07/2024-12:46:59] [I] [TRT] ----------------------------------------------------------------
[01/07/2024-12:46:59] [I] Finished parsing network model. Parse time: 0.0404419
[01/07/2024-12:46:59] [I] FP32 and INT8 precisions have been specified - more performance might be enabled by additionally specifying --fp16 or --best
[01/07/2024-12:46:59] [W] [TRT] Calibrator won't be used in explicit precision mode. Use quantization aware training to generate network with Quantize/Dequantize nodes.
[01/07/2024-12:46:59] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[01/07/2024-12:47:07] [E] Error[10]: Could not find any implementation for node SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add.
[01/07/2024-12:47:07] [E] Error[10]: [optimizer.cpp::computeCosts::4048] Error Code 10: Internal Error (Could not find any implementation for node SandGlass4.8.weight + /SandGlass4/SandGlass4.8/_weight_quantizer/QuantizeLinear + /SandGlass4/SandGlass4.8/Conv + /Add.)
[01/07/2024-12:47:07] [E] Engine could not be created from network
[01/07/2024-12:47:07] [E] Building engine failed
[01/07/2024-12:47:07] [E] Failed to create engine from model or file.
[01/07/2024-12:47:07] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v9200] # trtexec --onnx=mobilenetXT_qat.onnx --int8
One side question: you model is a QAT model, why do you provide a calibrator for it(PTQ)?
Yes, it was my mistake，The second problem was that I did not have a clear understanding of the quantization process at that time, and confused QAT with PTQ. Now I deleted the calibrator for PTQ and directly switched to trt.