Yuyi Ao comments

Repositories
Issues
Comments

Results 4 comments of


                                            Yuyi Ao

[Bug]: CUDA illegal memory access error when `enable_prefix_caching=True`

Is this issue solved in the latest version?

[Core] fix _get_num_new_tokens() for _schedule_default()

/ready

[Core] fix _get_num_new_tokens() for _schedule_default()

Hi @DarkLight1337, can you take a look at this change?

[Core] fix _get_num_new_tokens() for _schedule_default()

> can you add a unit test? are you saying if num_max_batched_tokens < number of decode requests, this happens? Thanks for reply, but I realize I was wrong. It comes...