cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Consider adopting a stateless PRNG API

Open alxmrs opened this issue 1 year ago • 3 comments

While I'm not familiar with the Philox pseudo-random number generator (PRNG) in Numpy (it does look well suited to generation in a distributed setting), I think adopting a stateless PRNG API will be useful long-term for cubed. In addition to working in a parallel/distributed setting, Cubed also has to consider how it can best perform computation with vectorization and hardware acceleration (#304, #490).

I'm quite persuaded by the design of Jax's PRNG system that statelessness (if not also splittable). I belive this approach will prove useful in the long-term.

https://github.com/google/jax/blob/main/docs/jep/263-prng.md

FWIW, I believe any ML framework will have to have special cases for randomness, given the constraints of hardware (GPUs/TPUs).

https://pytorch-dev-podcast.simplecast.com/episodes/random-number-generators

alxmrs avatar Jul 21 '24 16:07 alxmrs