Reproducibility over different machines

[sinemuysal]

Hello,

I am using cvxpylayer with PyTorch. I am running the same code with same seeds on different machines, and results are different from each other. Initial weights are same but the gradients from backward() function are different. It seems like this a problem in PyTorch [https://discuss.pytorch.org/t/reproducibility-over-different-machines/63047] but I want to double check if there is a randomness coming from cvxpylayer.

Also if you have any suggestions to mitigate this reproducibility problem, it would be great! Do you experience the same issue with tensorflow?

Thank you!

[sbarratt]

Have a few follow up questions: Is the architecture, OS, etc... of both machines the same (eg AMD vs Intel, and Ubuntu vs Mac OS)? How different are the gradients? Are the outputs from the cvxpylayer the same? Is the cvxpylayer called just by itself, or is the pre and/or post processing in pytorch?

[sinemuysal]

1. I have a MacBook Pro with macOS Big Sur version 11.2.3, the other machine is a university research cluster which has Linux operating system with Intel Skylake CPU nodes. Let me know if you need more info.

2. In each training, gradients differ by the order of 1e-9 at the initial steps but after 9 steps difference is in the order of 1e-4

3. Similar to weights, difference in the cvxpylayer outputs increases with the number of steps - to the range 1e-4.

4. There is one layer before cvxpylayer, and it’s parameters depends on the weights from the previous layer and initial inputs.