TensorRT-LLM icon indicating copy to clipboard operation
TensorRT-LLM copied to clipboard
verified publisher icon NVIDIA

refactor: Remove speculative decoding parameters from stateful decoders

Open Funatiq opened this issue 8 months ago • 15 comments

Simplify StatefulGptDecoderBatched constructor:

  • Remove speculative decoding mode parameter
  • Initialize with default mode=None
  • Update GptSession class accordingly

Simplify setup method signatures in StatefulGptDecoder and StatefulGptDecoderBatched:

  • Remove maxTokensPerStep parameter
  • Initialize decoders with default maxTokensPerStep=1
  • Update GptSession class accordingly

Funatiq avatar Mar 24 '25 10:03 Funatiq

/bot run

Funatiq avatar Mar 24 '25 10:03 Funatiq

PR_Github #290 [ run ] triggered by Bot

niukuo avatar Mar 24 '25 10:03 niukuo

PR_Github #290 [ run ] completed with state FAILURE /LLM/main/L0_MergeRequest_PR pipeline #279 completed with status: 'FAILURE'

niukuo avatar Mar 24 '25 12:03 niukuo

/bot run

Funatiq avatar Mar 24 '25 15:03 Funatiq

PR_Github #320 [ run ] triggered by Bot

niukuo avatar Mar 24 '25 15:03 niukuo

PR_Github #320 [ run ] completed with state SUCCESS /LLM/main/L0_MergeRequest_PR pipeline #299 completed with status: 'FAILURE'

niukuo avatar Mar 24 '25 17:03 niukuo

/bot run

Funatiq avatar Mar 25 '25 07:03 Funatiq

PR_Github #396 [ run ] triggered by Bot

niukuo avatar Mar 25 '25 07:03 niukuo

PR_Github #396 [ run ] completed with state FAILURE /LLM/main/L0_MergeRequest_PR pipeline #352 completed with status: 'FAILURE'

niukuo avatar Mar 25 '25 08:03 niukuo

/bot run

Funatiq avatar Mar 25 '25 09:03 Funatiq

PR_Github #421 [ run ] triggered by Bot

niukuo avatar Mar 25 '25 09:03 niukuo

PR_Github #421 [ run ] completed with state FAILURE /LLM/main/L0_MergeRequest_PR pipeline #363 completed with status: 'FAILURE'

niukuo avatar Mar 25 '25 11:03 niukuo

/bot run --disable-fail-fast

Funatiq avatar Mar 25 '25 18:03 Funatiq

PR_Github #462 [ run ] triggered by Bot

niukuo avatar Mar 25 '25 18:03 niukuo

PR_Github #462 [ run ] completed with state SUCCESS /LLM/main/L0_MergeRequest_PR pipeline #394 completed with status: 'SUCCESS'

niukuo avatar Mar 25 '25 22:03 niukuo

Looks good to me

DomBrown avatar Mar 26 '25 11:03 DomBrown

/bot reuse-pipeline

Funatiq avatar Mar 26 '25 12:03 Funatiq

PR_Github #583 [ reuse-pipeline ] triggered by Bot

niukuo avatar Mar 26 '25 12:03 niukuo

PR_Github #583 [ reuse-pipeline ] completed with state SUCCESS Reusing PR_Github #462 for commit 094c3de

niukuo avatar Mar 26 '25 12:03 niukuo