lightning-thunder icon indicating copy to clipboard operation
lightning-thunder copied to clipboard

Replacing uniform with uniform_philox , and incrementing PRNG state

Open kiya00 opened this issue 1 year ago • 2 comments

Before submitting
  • [ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • [ ] Did you read the contributor guideline, Pull Request section?
  • [ ] Did you make sure to update the docs?
  • [ ] Did you write any new necessary tests?

What does this PR do?

Fixes #114

  • [x] Add the prim operators to query/update the default CUDA RNG state PR
  • [ ] Fix #231
  • [x] Add transformation to replacing uniform with uniform_philox (this PR)
  • [ ] Make sure the rematerialization works properly for Dropout

kiya00 avatar Apr 19 '24 10:04 kiya00

TODO: fix #231 update the offset according to nvfuser properly

kiya00 avatar Apr 22 '24 09:04 kiya00

Let's split replace_uniform into a separate PR. Primitives for getting and setting the state can be tested without replace_uniform and having https://github.com/Lightning-AI/lightning-thunder/issues/231 fixed is not required for these primitives.

IvanYashchuk avatar Apr 24 '24 09:04 IvanYashchuk

Hi @mruberry @jjsjann123 @IvanYashchuk , I modified it according our design review discussion, could you take a look?

kiya00 avatar May 28 '24 12:05 kiya00

merged 9 commits into rng_update_prims from uniform_rng

We had a manual stack here 😃

IvanYashchuk avatar May 29 '24 18:05 IvanYashchuk

Yeah, I'll learn to read...

t-vi avatar May 29 '24 18:05 t-vi

sorry for the trouble, I should get that draft status on

kiya00 avatar May 29 '24 18:05 kiya00