Po-Wei (Vincent) comments

Results 70 comments of


                                            Po-Wei (Vincent)

Fail early if output engine file path is unwritable in trtexec

Hello @PierreMarieCurie thanks for your contribution! Can you target the main branch and have your commits rebase on that?

Fail early if output engine file path is unwritable in trtexec

Thanks Pierre for the fast response! I lost my permission to trigger the pipeline but should be able to run next week. Will lyk with the pipeline results.

Fail early if output engine file path is unwritable in trtexec

/blossom-ci

Fail early if output engine file path is unwritable in trtexec

/blossom-ci

use selected index past past key value in attention when using contin…

Closing since no response after https://github.com/NVIDIA/TensorRT-LLM/pull/2682#issuecomment-2761036095. Feel free to reopen! @Eayne

`configurePlugin` is called repeatedly for my BF16 SDPA plugin — how to run initialization graph only once?

@KarlDe1 `configurePlugin` is not designed to do one time initializations, as it will be called if outputs like `getOutputDimensions()`, `supportsFormatCombination()`, `getOutputDataType()` has changed. Can you take a look at output...

`configurePlugin` is called repeatedly for my BF16 SDPA plugin — how to run initialization graph only once?

1. I don't think that's expected. I can't repro the issue effectively though without an example. 2. This is the only way now. At least for usecases that I'm aware,...

Straightforward way to cancel `GenerationSession`

Hello @daniel-wispr ! Is this issue still relevant?

Int8 quantized model performs worse or similar to non quantized fp32 or fp16 model.

Closing due to inactive. Please feel free to reopen!

A Simple Engine Build Fail on TensorRT 8.6

Able to build the engine with `trtexec --onnx=debug.onnx --saveEngine=out.engine` with the latest TRT version. Feel free to reopen if you still see the issue with the latest TRT version!