asedova

@oschuett it seems to still be an issue, but I need to rebuild with the latest version to double check. However, I am wondering if this is related to a...

1. A SIMULATED MD trajectory in digital/numerical world can totally be deterministic if it is coded that way. It's using RNGs! You can set the seeds. There is nothing inherently...

> Our customized CUDA OP also uses non-deterministic [`atomicAdd`](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html?highlight=atomicAdd#atomicadd). > > https://github.com/deepmodeling/deepmd-kit/blob/b875ea8f6661b6e1567537ead7e2b4a8b14ea113/source/lib/src/gpu/prod_force.cu#L73 > > The deterministic implementation may need extra effort, which might not be worth doing. Yep--already found this...

I would like to measure how much the custom CUDA kernel contributes, and how much any TF ops contribute. I am wondering if there is a way to use GPU...

By the way, there seems to be another source of "non-determinism" in the deepmd code that may actually be a bug. I ran the `se_e2_a` water example a bunch of...

learning rate: Number of steps were the same, nothing was different except I ran it again, I am pretty sure. I will run some more tests to verify. What I...

So I've done some reproducibility testing just on model training and inference. I ran the exact same training on the same data, same hyperparameters, twice to get 2 "identical" models...

Yes, there should be some nondeterminism with TF. But I didn't expect it to affect the forces THAT much. That's a lot. And it seems strange that it would affect...

I'm also wondering what it would take to turn on TF determinism in DeePMD. Some detailed notes on doing this can be found here: https://github.com/NVIDIA/framework-reproducibility/blob/master/doc/d9m/tensorflow.md We are working with Duncan/NVIDIA...

Yes, that is what I am talking about. Where in the code would be the top-most entrypoint to add this command so it propagates down to all the TF calls?...