q yao

icon indicating a website link https://grimoire.github.io/ icon indicating an email [email protected]

icon location Best LLM inference engine: https://reurl.cc/aLxR2Y

Results 318 comments of q yao

converted bart model is slower than the original one during inference time

Sorry, I still do not have any solution.

converted bart model is slower than the original one during inference time

And, by the way, have you tried to increase `max_workspace_size`? Some tactic need more workspace to perform accelerate.

converted bart model is slower than the original one during inference time

I set the `max_workspace_size=1

Hi, how to solve it with list output?

What is the output structure of your model? The convertor only accept tensor or list of tensors as output. Nested list is not convertable as the final output.

WARNING: Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are: (+ (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (+ (# 3 (SHAPE input_0)) -1) 2) 2) 2) 2) 2) 1) (+ (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (+ (# 2 (SHAPE input_0)) -1) 2) 2) 2) 2) 2) 1)

The warning about myelin happened when the model is too complex and too many shape inference need to be performed. Try to avoid dynamic shape (you can set the same...

WARNING: Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are: (+ (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (+ (# 3 (SHAPE input_0)) -1) 2) 2) 2) 2) 2) 1) (+ (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (FLOOR_DIV (+ (# 2 (SHAPE input_0)) -1) 2) 2) 2) 2) 2) 1)

It is hard to say, the warning means there is an unsupported layer in the network. The output of this node would be treated as constant value. So if these...

load libtorch2trt_dynamic.so error

`--plugins` flag is not used in this repo. Please install follow the README of this repo, not the official one.

load libtorch2trt_dynamic.so error

Please follow [these](https://github.com/grimoire/torch2trt_dynamic#setup) steps. you do **NOT** need to change anything inside setup.py, The plugin used in this repo is [amirstan_plugin](https://github.com/grimoire/amirstan_plugin).

load libtorch2trt_dynamic.so error

Yes, read [this](https://github.com/grimoire/mmdetection-to-tensorrt/blob/master/docs/getting_started.md#use-in-c) for detail.

Build docker image with amirstan plugin on actions

Sounds like a good way to add unit tests. Thanks, I will try this weekend.