DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

fix: remove layer_past storing in DeepSpeedTransformerInference

Open codertimo opened this issue 2 years ago • 1 comments

What Changed?

  • this PR fix #1925
  • remove layer_past storing in transformer_inference

why?

  • layer_past should not be stored in model. It should given by input.
  • layer_past never released. If model is repeatly called, layer_past constantly growing. It's not what we expected.

codertimo avatar May 01 '22 12:05 codertimo

Can one of the admins verify this patch?

rocm-mici avatar Jun 09 '22 20:06 rocm-mici

i wrap model with deepspeep after using huggingface transfomers to load model firstly, the i met

Floating point exception (core dumped)

this occur with some sentence as input, not all the sentences

520jefferson avatar Apr 03 '23 16:04 520jefferson

Hi @codertimo - Do you believe this PR is still useful? I'm not able to resolve the conflicts on your fork with our master branch. If so, could you resolve these conflicts, and we can review. Apologies for the delay on reviewing in the first place.

loadams avatar Aug 23 '23 16:08 loadams