Lev Kurilenko
Lev Kurilenko
Hi @hyhuang00, I believe this issue may have been resolved with [PR 2212](https://github.com/microsoft/DeepSpeed/pull/2212) in the DeepSpeed repo. Can you please try installing the latest version of DeepSpeed and running again...
Hello @Jirigesi, Apologies for the delayed follow up to your issue. The [inference tutorial](https://www.deepspeed.ai/tutorials/inference-tutorial/#loading-checkpoints) is slightly out of date with the code. For checkpoint loading to work using a `checkpoint.json`...
Can you please try using the [inference-test.py](https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/text-generation/inference-test.py) example from the [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/inference/huggingface/text-generation#deepspeed-huggingface-text-generation-examples) repo? The command to use should be: ``` deepspeed --num_gpus 8 inference-test.py --name microsoft/bloom-deepspeed-inference-int8 --ds_inference --use_kernel --use_meta_tensor --replace_method=auto --dtype...
Kicked off a manual run of the `nv-ds-chat` GH workflow since this PR modifies the Hybrid Engine: https://github.com/microsoft/DeepSpeed/actions/runs/9199337454 @loadams, @jomayeri, FYI.
> Hi @lekurile , can you please run ds-chat coverage on this PR? i reproduced the issue was reproted in [nelyahu@bc48371](https://github.com/nelyahu/DeepSpeed/commit/bc48371c5e1fb8fd70fc79285e66201dbb65679b) and fixed it. would like to get a pre-commit...
> > > Hi @lekurile , can you please run ds-chat coverage on this PR? i reproduced the issue was reproted in [nelyahu@bc48371](https://github.com/nelyahu/DeepSpeed/commit/bc48371c5e1fb8fd70fc79285e66201dbb65679b) and fixed it. would like to get...
> Hi @sywangyi, thanks for the PR. I tested this on my side and it looks good. We may just want to move these changes to another file since they...
Hi @brevity2021, Meta tensor checkpoint loading is only supported when kernel injection is enabled. Please provide the `--use_kernel` argument to the `inference-test.py` script when running this example and using meta...
> Hey @lekurile, If you have some time can you please review this. Hi @satpalsr, Thank you for the contribution! I left a review comment about moving the check out...
> Hey @lekurile, > > Thanks for checking, but I somehow can't find your review. As I understand you are suggesting to move the function out. Will make changes. Thanks....