NeuralPDE.jl
NeuralPDE.jl copied to clipboard
`discretization.iteration` is not thread-safe for reuse with additional discretizations
I'm sure @zoemcc noticed this when doing the experiment manager, but basically what happens is that there's a single discretization.iteration
value that is then used in the loss functions, and so if you generate two separate losses from the same symbolic_discretize
you will share the iterations and thus have both threads increment the same counter. I don't think this is a major deal since this is still safe with distributed computing since that will require sending the function to a new heap with thus a new counter, and thus it's only threading not distributed that can hit this issue, and almost certainly with PINNs your multithreading at the BLAS level so multithreading is a bad idea anyways, but just documenting this in case anyone sees oddities in the iterations counter knows what it is.
I don't plan to fix it right now since the surely the answer to any case that hits this is "why are you not doing distributed?", but if anyone has a valid use case for it then we should find a way to clean this up by defining the iterations at the function generation stage not the symbolic stage.