tokio-rs
Document determinism guidelines
Turmoil is built on the concept of deterministic execution. Using structures such as HashMap initialize with non-deterministic RandomState. Both the internals of turmoil and applications using it need to buy in.
e.g. HashMap, HashSet, tokio::select!, etc.
Document the guidelines.
Here are some I ran into in my own testing:
- Make sure your servers are each using their own RNG seed. Otherwise, they will behave identically and often fail to make progress, elect a leader, etc.
- Be wary of libraries that might include randomness. For example, many client libraries contain "exponential backoff with random jitter". You'll need to make sure those RNGs are properly seeded.
-
Be wary of hashmaps. Initialize hashmaps with a deterministic hasher, or use ones with deterministic iteration order (
indexmap)- This one may be hard to accomplish because hashmaps are ubiquitous in library code.
Our experience jives with @dtwitty points. Ultimately overriding certain randomness & time-related libc fns proved necessary to ensure perfect determinism (shared some thoughts here https://x.com/shikhrr/status/1818476801657598128)
To be specific, we override: clock_gettime, getrandom, and getentropy (just uses getrandom). Simply having those symbols defined in DST mode is sufficient to override libc, as noted by the author of madsim here https://github.com/madsim-rs/madsim/discussions/159
