`Torch.compile()` and activation checkpointing are incompatible

[dirkgr]

Activation checkpointing needs to keep track of the state of the random number generator, which fails with torch.compile(). Rumor has it that the latest torch nightly has this fixed, so we should try this.

[dirkgr]

https://github.com/pytorch/pytorch/issues/97436

[dumitrac]

Marking the items prior to Feb 29th as "closed".