Pierre Janeke comments

Repositories
Issues
Comments

Results 5 comments of


                                            Pierre Janeke

compatibility issues and memory leak problems --enable-flashinfer

I see 2. was fixed with https://github.com/sgl-project/sglang/commit/b0890631a011be28d5ef5a0b4d5551fdeb94ab25

compatibility issues and memory leak problems --enable-flashinfer

Does this mean the problem with 1. is fixed @merrymercy?

[QUESTION]: The param:output_fields in the search method is invalid

Any progress on this?

Support for Constrained decoding

@rlouf did you manage to make much progress yet?

kv cache pool leak detected when benchmark llama13B-awq using A40

I had a similar problem running on an EC2 g5.2xlarge instance (1 x A10G) using openchat/openchat3.5-0106. I have long sequences (6-7k tokens). A batch size of 19 sequences is fine,...