NeuralPDE.jl icon indicating copy to clipboard operation
NeuralPDE.jl copied to clipboard

`discretization.iteration` is not thread-safe for reuse with additional discretizations

Open ChrisRackauckas opened this issue 2 years ago • 0 comments

I'm sure @zoemcc noticed this when doing the experiment manager, but basically what happens is that there's a single discretization.iteration value that is then used in the loss functions, and so if you generate two separate losses from the same symbolic_discretize you will share the iterations and thus have both threads increment the same counter. I don't think this is a major deal since this is still safe with distributed computing since that will require sending the function to a new heap with thus a new counter, and thus it's only threading not distributed that can hit this issue, and almost certainly with PINNs your multithreading at the BLAS level so multithreading is a bad idea anyways, but just documenting this in case anyone sees oddities in the iterations counter knows what it is.

I don't plan to fix it right now since the surely the answer to any case that hits this is "why are you not doing distributed?", but if anyone has a valid use case for it then we should find a way to clean this up by defining the iterations at the function generation stage not the symbolic stage.

ChrisRackauckas avatar Jun 29 '22 10:06 ChrisRackauckas