Maybe remove calls to `set.seed` and `torch_manual_seed()` from the tests

[sebffischer]

If we set the seeds in the tests, everything afterwards is deterministic. For some tests that randomly generate data, it would maybe be better to have them be executed randomly to increase the test coverage.

[dfalbel]

This will be a significant amount of work to implement given how it currently expects.

[sebffischer]

What I mean with the issue is that one could instead using something like with_torch_local_seed(...) to make the subsequent tests not deterministic.