Race condition in ensemble simulation with callbacks?

[dcabecinhas]

I reliably get this error about Tried to add a tstop that is behind the current time. This is strictly forbidden when running the threaded defaults.

The code runs without errors if using EnsembleSerial().

julia> include("test.jl")
ERROR: LoadError: TaskFailedException:
Tried to add a tstop that is behind the current time. This is strictly forbidden
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] add_tstop! at /Users/dcabecinhas/.julia/packages/OrdinaryDiffEq/yn1YB/src/integrators/integrator_interface.jl:96 [inlined]
 [3] (::DiffEqCallbacks.var"#45#50"{typeof(periodic!),Float64,...

Code for replication

using DifferentialEquations

function f!(du,u,p,t) 
  du .= -1.01*u
end

u0 = rand(1000)
tspan = (0.0,100.0)
prob = ODEProblem(f!,u0,tspan)

function prob_func(prob,i,repeat)
  prob
end

function periodic!(integrator) end
cb = PeriodicCallback(periodic!, 0.1)

ensemble_prob = EnsembleProblem(prob,prob_func=prob_func)
ensemble_sol = solve(ensemble_prob,
                      Tsit5(),
                      # EnsembleSerial(),
                      trajectories=100,
                      callback = cb,
                      );

[ChrisRackauckas]

The issue is that the PeriodicCallback holds state, so it's not thread safe. If it's done per-problem it's fine:

using DifferentialEquations

function f!(du,u,p,t)
  du .= -1.01*u
end

u0 = rand(1000)
tspan = (0.0,100.0)

function periodic!(integrator) end
cb = PeriodicCallback(periodic!, 0.1)

prob = ODEProblem(f!,u0,tspan,callback=cb)

function prob_func(prob,i,repeat)
  prob
end


ensemble_prob = EnsembleProblem(prob,prob_func=prob_func)
ensemble_sol = solve(ensemble_prob,
                      Tsit5(),
                      # EnsembleSerial(),
                      trajectories=100,
                      );

This is tricky to fix in general, and suggests that we should mostly document to associate callbacks with problems (which in many ways is more natural)

[dcabecinhas]

Thanks for the explanation. Indeed, the documentation could be better.

Most (if not all) examples in the Callbacks section associate the callbacks with solve() and in the Parallel Ensemble section the only mention to callbacks is this one:

Note: If the problem has callbacks, the functions for the condition and affect! must be named functions (not anonymous functions).