🐛 [Bug] Torch-TRT has more Reformatting than ONNX-TRT

[zewenli98]

Bug Description

Torch-TRT has more Reformatting than ONNX-TRT, which affects perf. For example:

, { "name" : "Reformatting CopyNode for Input Tensor 1 to [CONVOLUTION]-[aten_ops.convolution.default]-[model.0.residual/convolution] + [ELEMENTWISE]-[aten_ops.add.Tensor]-[model.0/add]", "timeMs" : 2.19312, "averageMs" : 0.0190706, "medianMs" : 0.01872, "percentage" : 0.183659 }
, { "name" : "Reformatting CopyNode for Input Tensor 1 to PWN([PARAMETRIC_RELU]-[aten_ops._prelu_kernel.default]-[model.1.submodule.0.conv.unit0.adn.A/_prelu_kernel_2])", "timeMs" : 1.1616, "averageMs" : 0.0101009, "medianMs" : 0.003072, "percentage" : 0.0972762 }
, { "name" : "Reformatting CopyNode for Input Tensor 1 to PWN([PARAMETRIC_RELU]-[aten_ops._prelu_kernel.default]-[model.1.submodule.0.conv.unit1.adn.A/_prelu_kernel_3])", "timeMs" : 0.3312, "averageMs" : 0.00288, "medianMs" : 0.003072, "percentage" : 0.0277357 }
, { "name" : "Reformatting CopyNode for Input Tensor 1 to [CONVOLUTION]-[aten_ops.convolution.default]-[model.1.submodule.0.residual/convolution_3] + [ELEMENTWISE]-[aten_ops.add.Tensor]-[model.1.submodule.0/add_1]", "timeMs" : 0, "averageMs" : 0, "medianMs" : 0, "percentage" : 0 }