996_icu comments

Results 5 comments of


                                            996_icu

[Bug] Vision attention mask cache is never released and cause OOM

By the way, I want to know that I see `VisionBlock` uses `VisionSdpaAttention` instead of `VisionTritonAttention` by default, is there any problem in VisionTritonAttention? And I find that in `VisionSdpaAttention`,...

[Bug] Vision attention mask cache is never released and cause OOM

> > By the way, I want to know that I see `VisionBlock` uses `VisionSdpaAttention` instead of `VisionTritonAttention` by default, is there any problem in VisionTritonAttention? > > And I...

[Bug] Vision attention mask cache is never released and cause OOM

> I didn't make it very clear previously, my apologies. Let me clarify it: > > ## Transformers Qwen > > The official modeling_qwen2_vl.py decides the attention for ViT based...

[Bug] RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 458752000 and alignment 16 in AlignedAllocator

> @josephydu Can you try it again with sglang v0.3.1.post3? > > I run the same command on 8xH100 and did not find any issues. I still got the problem...

[Bug] RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 458752000 and alignment 16 in AlignedAllocator

> @merrymercy I am also getting the same issue when running llama 405B FP8 from neuralmagic on 8h100s > > This is how I launch the server: python3 -m sglang.launch_server...