sroy745 issues

Repositories
Issues
Comments

Results 2 issues of


                                            sroy745

[Speculative Decoding] Enabling bonus token in speculative decoding for KV cache based models

FILL IN THE PR DESCRIPTION HERE FIX #4212 In this PR we make the following changes 1. Update the spec_decode_worker to keep track of the sequence_ids which we were assigned...

[Encoder decoder] Add cuda graph support during decoding for encoder-decoder models

In this PR we are adding support for CUDA Graph capture & replay during the decoding phase for encoder-decoder models. Currently this support is missing for encoder-decoder models. To that...

ready