Mor Zusman comments

Results 10 comments of


                                            Mor Zusman

[New Model]: Jamba (MoE Mamba from AI21)

#4115

[BUG] Zeros in returned k_v cache when get_present = True

Opened a PR fixing this issue https://github.com/microsoft/DeepSpeed/pull/2828

[BUG] DeepSpeed Inference reports Signal code: Integer divide-by-zero when Seq length is 4096 for GPT2

@zhen-jia I also encountered this bug, fixed it by simply changing ATTN_THREADS to 512

[Model] Jamba support

> May I get an update regarding the status of this PR? It seems the author stopped working on it? We're currently still working on it, The PR works well,...

[Model] Jamba support

AFAIU CI distributed-tests-2-gpus test fails regardless of this PR.

[Model] Jamba support

> QQ: Does this PR support parallel sampling (i.e., `n` > 1 in sampling params)? While I don't think it is not necessary to support parallel sampling in this PR,...

[Model] Jamba support

Tests failed due to timeouts to HF Ready to be merged

feat: Initial state support for Mamba SSM (1)

@daphneOdera-618 Yeah, the default setup.py behaviour is to download the upstream's wheel upon "installing", What you would need to do to force build is to add `MAMBA_FORCE_BUILD=TRUE pip install .`

[BugFix] get_and_reset only when scheduler outputs are not empty

@simon-mo Are there any blockers for merge? Thanks

[Bug]: Mamba should return states in fp32

@toslunar In tests we've done with bf16 , we haven't found evidence for quality degradation, can you share some results or benchmarks?