DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Rearrange inference OPS and stop using builder.load

Open oelayan7 opened this issue 1 year ago • 10 comments

This PR mainly handles all places where InferenceBuilder is used to access any op or a specific implementation for an op. Instead an op is defined, and its proper implementation is picked inside and the usage will be transparent to the user. What was done in the PR:

  1. Added missing ops (added a py file with fallback mechanism)
  2. Added missing fallback implementations for existing ops
  3. removed all usages for builder.load and replaced them with ops instead.
  4. added workspace op and inferenceContext which contains all workspace related functions and inferenceContext is the python fallback of inferenceContext in CUDA
  5. a small change to softmax_context signature to fit the fallback signature.

oelayan7 avatar May 01 '24 09:05 oelayan7

Hi @oelayan7 - thanks for the contribution, could you take a look at the failing tests?

loadams avatar May 22 '24 21:05 loadams

Kicked off a manual run of the nv-ds-chat GH workflow since this PR modifies the Hybrid Engine: https://github.com/microsoft/DeepSpeed/actions/runs/9199337454

@loadams, @jomayeri, FYI.

lekurile avatar May 22 '24 22:05 lekurile

@loadams @lekurile it is fixed now, can you please trigger the unit tests?

oelayan7 avatar Jun 10 '24 12:06 oelayan7

@loadams @lekurile it is fixed now, can you please trigger the unit tests?

Hi @oelayan7, kicked off the unit tests.

lekurile avatar Jun 10 '24 16:06 lekurile

@loadams Can you please trigger the CI (hopefully the last time :) )

oelayan7 avatar Jun 19 '24 09:06 oelayan7

@loadams Can you please trigger the CI (hopefully the last time :) )

@oelayan7 - CI failures appear to be real for a few things if you could take a look

loadams avatar Jun 26 '24 20:06 loadams

@loadams Can you please trigger the CI (hopefully the last time :) )

@oelayan7 - CI failures appear to be real for a few things if you could take a look

@loadams you were right, there were two minor issues and they were fixed.

oelayan7 avatar Jun 30 '24 05:06 oelayan7

@loadams @arashb @tjruwase @mrwyattii @awan-10 Can anyone trigger the unit-tests please?

oelayan7 avatar Jul 11 '24 05:07 oelayan7

@loadams @arashb @tjruwase @mrwyattii @awan-10 Can anyone trigger the unit-tests please?

Yes, though we have a known issue in CI now, so some tests will still not run.

loadams avatar Jul 11 '24 16:07 loadams

@loadams @arashb @tjruwase @mrwyattii @awan-10 Can anyone trigger the unit-tests please?

Yes, though we have a known issue in CI now, so some tests will still not run.

@oelayan7 - we've fixed CI now, your PR should be able to be merged after a successful review

loadams avatar Aug 20 '24 17:08 loadams

@oelayan7 - I've got to push a fix for nv-accelerate tests, will get this merged shorlty.

loadams avatar Sep 25 '24 22:09 loadams