Olatunji Ruwase

Results 627 comments of Olatunji Ruwase

[REQUEST] Asynchronous Checkpointing

> Will [FastPersist](https://arxiv.org/abs/2406.13768) be open-sourced in the next DeepSpeed release ? @cailun01, yes, we plan to open-source soon.

[REQUEST] Asynchronous Checkpointing

@Irene-123, you are welcome to give it a try. But I suspect this requires non-trivial effort and probably not a good first issue. @zaptrem, are you able to provide guidance...

[BUG] Universal checkpoint conversion failed

Closing due to lack of activity. Please re-open as needed.

DeepSpeed 2 GPU slower than 1 GPU. PyTorch DDP - much faster. Why?

@Mars2018, @slchenchn, @gray311 can you please provide repro steps? Thanks!

when I finetune the model use deepspeed on 2 4*A800s，log only contain worker1

@bill4689, do you know which code is generating those outputs? I don't believe it is DeepSpeed because DeepSpeed is unaware of epochs. Can you please try to locate the source...

when I finetune the model use deepspeed on 2 4*A800s，log only contain worker1

Closing for lack of response. Please feel to re-open as needed.

[BUG] oneapi/ccl.hpp: No such file or directory.

@delock, can you help with this?

lr scheduler defined in config cannot be overwritten by lr scheduler defined in code and pass to `deepspeed.initialize` [BUG]

@xiyang-aads-lilly, this is a good catch. Your proposed solution looks reasonable. Are you able to provide a PR? Thanks!

[BUG]RuntimeError: disagreement between rank0 and rank1: rank0:

@eonsparks, thanks for suggesting solutions!

In distributed training, in order to continue training, an error occurred when loading model checkpoints after saving them.

@WhaleSpring, can you clarify two things to help debugging. 1. Are checkpoints saved onto local disk? The load logs contain `/tmp/` references which is typically a node local storage. 2....