Take awhile to precompile with style all of the DiffEq stack, no denial

Our goal is to get the entire DiffEq stack compile times down. From now on we should treat compile time issues and compile time regressions just like we treat performance issues, 😢 😿 😭 and then fix them. For a very long time we did not do this because, unlike runtime performance, we did not have a good way to diagnose the causes, so it was 🤷 whatever we got is what we got, move on. But now, thanks to @timholy, we have the right tools and have learned how to handle compile time issues in a very deep way so the game is on.

Our goal is to get compile times of all standard workflows to at least 0.1 seconds. That is quick enough that you wouldn't care all that much in interactive usage, but still not an unreasonable goal given the benchmarks. This issue is how to get there and how the community is can help. This will cover:

What have we done so far (so you can learn from our experience)
How much has it mattered and to what use cases
How can a user give helpful compile time issues
What are some helpful tricks and knowledge to share?
What are some of the next steps

Let's dig in.

What were our first strides?

We have already made a great leap forward in the last week+ since the JuliaCon hackathon. The very long tl;dr i in https://github.com/SciML/DiffEqBase.jl/pull/698 . However, that doesn't capture the full scope of what was actually done:

https://github.com/SciML/OrdinaryDiffEq.jl/pull/1460
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1465
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1467
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1468
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1469
https://github.com/SciML/DiffEqBase.jl/pull/688
https://github.com/SciML/DiffEqBase.jl/pull/696
https://github.com/SciML/DiffEqBase.jl/pull/697
https://github.com/SciML/DiffEqBase.jl/pull/698
https://github.com/SciML/SciMLBase.jl/pull/95
https://github.com/JuliaDiff/SparseDiffTools.jl/pull/147
https://github.com/JuliaDiff/SparseDiffTools.jl/pull/149
https://github.com/YingboMa/RecursiveFactorization.jl/pull/29
https://github.com/YingboMa/RecursiveFactorization.jl/pull/30
https://github.com/JuliaSIMD/TriangularSolve.jl/pull/8
https://github.com/SciML/OrdinaryDiffEq.jl/pull/1470 (about to merge after a few more fixes)

with some bonus PRs and issues like:

https://github.com/JuliaGraphs/LightGraphs.jl/pull/1581
https://github.com/JuliaDiff/DiffRules.jl/pull/64
https://github.com/JuliaDebug/Cthulhu.jl/issues/184
https://github.com/JuliaDebug/Cthulhu.jl/pull/185
https://github.com/JuliaLang/julia/issues/41750
https://github.com/JuliaLang/julia/pull/41813

Show me some results

The net result is something like this. On non-stiff ODEs, compile times dropped from about 5 seconds to sub 1 second, and on stiff ODEs compile times dropped from about 22 seconds to 2.5 seconds. The tests are things like:

https://github.com/SciML/OrdinaryDiffEq.jl/pull/1465

using OrdinaryDiffEq, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Tsit5()
tinf = @snoopi_deep solve(prob,alg)
itrigs = inference_triggers(tinf)
itrig = itrigs[13]
ascend(itrig)
@time solve(prob,alg)

v5.60.2
InferenceTimingNode: 1.249748/4.881587 on Core.Compiler.Timings.ROOT() with 2 direct children

Before
InferenceTimingNode: 1.136504/3.852949 on Core.Compiler.Timings.ROOT() with 2 direct children

Without `@turbo`
InferenceTimingNode: 0.956948/3.460591 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@inbounds @simd`
InferenceTimingNode: 0.941427/3.439566 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@turbo`
InferenceTimingNode: 1.174613/11.118534 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@inbounds @simd` everywhere
InferenceTimingNode: 0.760500/1.151602 on Core.Compiler.Timings.ROOT() with 2 direct children

# Today, a week after that PR
InferenceTimingNode: 0.634172/0.875295 on Core.Compiler.Timings.ROOT() with 1 direct children 🎉 
(it automatically does the emoji because the computer is happy too)

You read this as, it used to take 1.25 seconds for inference and 4.88 seconds for compilation in full, but now it's 0.63 and 0.88.

and https://github.com/SciML/DiffEqBase.jl/pull/698

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)

using OrdinaryDiffEq, SnoopCompile
prob = ODEProblem(lorenz,u0,tspan)

alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

After basic precompilation (most of the above PRs):
InferenceTimingNode: 1.460777/16.030597 on Core.Compiler.Timings.ROOT() with 46 direct children

After fixing precompilation of the LU-factorization tools:
InferenceTimingNode: 1.077774/2.868269 on Core.Compiler.Timings.ROOT() with 11 direct children

So that's the good news. The bad news.

The precompilation results do not always generalize

Take for example https://github.com/SciML/DifferentialEquations.jl/issues/785:

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

InferenceTimingNode: 1.535779/13.754596 on Core.Compiler.Timings.ROOT() with 7 direct children

"But Chris, I thought you just said that was sub 3 seconds, not 13.75 seconds compile time!". Well, that's with using OrdinaryDiffEq instead of using DifferentialEquations. And we see this when DiffEqSensitivity.jl or DiffEqFlux.jl gets involved.

So compile times are "a lot better"*, and the "+* are right now necessary when saying that. We need to fix that aspect of it.

How can I as a user help?

Good question, thanks for asking! Sharing profiles is extremely helpful. Take another look at https://github.com/SciML/DiffEqBase.jl/pull/698#issuecomment-895152646 . What was ran was:

using OrdinaryDiffEq, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

using ProfileView
ProfileView.view(flamegraph(tinf))

What this was saying was that the vast majority of the compile time was because the DEFAULT_LINSOLVE calling RecursiveFactorization.jl was not precompiling. Since we use our own full Julia-based BLAS/LAPACK stack, that gave a full 13 seconds of compilation since it would compile RecursiveFactorization.jl, TriangularSolve.jl, etc. in sequence on each first solve call of a session. This allowed us to identify the issue can create a tizzy of PRs that finally made that get cached.

If you check the DifferentialEquations.jl compile times, you'll see that part is back. Why won't it compile? Well that's a harder question, discussed in https://github.com/SciML/DiffEqBase.jl/pull/698#issuecomment-896984234, with the fix in https://github.com/SciML/DiffEqBase.jl/pull/698#issuecomment-897188008, but apparently gets invalidated. If that all doesn't make sense to you, that's fine! But if you can help us narrow in on what the real issues are, that will help us immensely.

And of course... you can always give us a sponsor and star the repos to help 😅 . But seriously though, I hope we can start using SciML funds to start scouring our repos and get a lot of people fixing compile time issues. More on that soon... very soon... 🤐

What are some helpful tricks and knowledge to share?

Yeah, what were our tricks? Well the big one is forcing compilation in using calls through small solves. In many cases compilation is solved by just putting a prototype solve inside of the package itself. For example, take a look at the precompile section of OrdinaryDiffEq.jl (https://github.com/SciML/OrdinaryDiffEq.jl/blob/v5.61.1/src/OrdinaryDiffEq.jl#L175-L193)

let
    while true
      function lorenz(du,u,p,t)
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
      end
      lorenzprob = ODEProblem(lorenz,[1.0;0.0;0.0],(0.0,1.0))
      solve(lorenzprob,Tsit5())
      solve(lorenzprob,Vern7())
      solve(lorenzprob,Vern9())
      solve(lorenzprob,Rosenbrock23())(5.0)
      solve(lorenzprob,TRBDF2())
      solve(lorenzprob,Rodas4(autodiff=false))
      solve(lorenzprob,KenCarp4(autodiff=false))
      solve(lorenzprob,Rodas5())
      break
    end
  end

Lorenz equation takes nanoseconds to solve, so we take this equation and solve it a few times at using time, which will then trigger precompilation of as many functions hit in that call stack as Julia will allow for. For some things, the reason why they are not precompiled is simply because they are never called during using time, so a quick fix for many of the compile time issues is to simply add a short little statement like this to using time and then Julia will cache its results. For everything else, there's Mastercard, and it'll be much more costly to solve, so those will need issues and the experts, and possibly some Base compiler changes. But we should at least grab all of the low hanging fruit ASAP.

This is actually a necessary condition for getting precompilation, since if a method is never called in using then it will never precompile, so this is a first step among many.

A major part of the solution was avoiding codegen when unnecessary. If you take a look at https://github.com/SciML/OrdinaryDiffEq.jl/pull/1465, you'll see things like:

@muladd function perform_step!(integrator, cache::Tsit5Cache{<:Array}, repeat_step=false)
  @unpack t,dt,uprev,u,f,p = integrator
  uidx = eachindex(integrator.uprev)
  @unpack c1,c2,c3,c4,c5,c6,a21,a31,a32,a41,a42,a43,a51,a52,a53,a54,a61,a62,a63,a64,a65,a71,a72,a73,a74,a75,a76,btilde1,btilde2,btilde3,btilde4,btilde5,btilde6,btilde7 = cache.tab
  @unpack k1,k2,k3,k4,k5,k6,k7,utilde,tmp,atmp = cache
  a = dt*a21
  @inbounds @simd ivdep for i in uidx
    tmp[i] = uprev[i]+a*k1[i]
  end
  f(k2, tmp, p, t+c1*dt)
  @inbounds @simd ivdep for i in uidx
    tmp[i] = uprev[i]+dt*(a31*k1[i]+a32*k2[i])
  end
...

i.e. new solver dispatches specifically for Array, to avoid dispatches like:

@muladd function perform_step!(integrator, cache::Tsit5Cache, repeat_step=false)
  @unpack t,dt,uprev,u,f,p = integrator
  @unpack c1,c2,c3,c4,c5,c6,a21,a31,a32,a41,a42,a43,a51,a52,a53,a54,a61,a62,a63,a64,a65,a71,a72,a73,a74,a75,a76,btilde1,btilde2,btilde3,btilde4,btilde5,btilde6,btilde7 = cache.tab
  @unpack k1,k2,k3,k4,k5,k6,k7,utilde,tmp,atmp = cache
  a = dt*a21
  @.. tmp = uprev+a*k1
  f(k2, tmp, p, t+c1*dt)
  @.. tmp = uprev+dt*(a31*k1+a32*k2)
...

The reason is that compile time profiling showcased that the major contributor was these https://github.com/YingboMa/FastBroadcast.jl code generation steps. Base Broadcast too is a major contributor to compile times. So at least on the pieces of code that 99% of users are using, we just expanded them out by hand, forced them to precompile, and that gave the 5 seconds to 1 second compile time change. I wouldn't say I am recommending you shouldn't use broadcast, but this is something useful to keep in mind. Broadcast is more generic, so you have to pay in compile time to use it. If you're willing to maintain the code, doing if u isa Array or making a separate dispatch can remove that factor.

Map is also a major contributor. We actually knew this was the case already, since there was a map call which changed Pumas compile times on a small example from 3 seconds to over 40 (https://github.com/SciML/SciMLBase.jl/pull/45). Removing map calls factored into these changes as well (https://github.com/JuliaDiff/SparseDiffTools.jl/pull/149). Again, this is something that could/should be handled at the Base compiler level, but we wanted as much of a fix ASAP so this was a nice and easy change which was within our power to ship in a week, and you can do the same.

The last thing, and the major step forward, was https://github.com/SciML/DiffEqBase.jl/pull/698#issuecomment-897188008 . As it explains, using a function barrier can cause inference to not know what functions it will need in a call, which makes Julia seem to compile a whole lot of junk. That can probably get fixed at the Base compiler level to some extent, as the example there was spending 13 seconds compiling (::DefaultLinSolve)(::Vector{Float64}, ::Any, ::Vector{Float64}, ::Bool) junk which nobody could ever call, since any call to that function would specialize on the second argument (a matrix type) and so it should've been compiling (::DefaultLinSolve)(::Vector{Float64}, ::Matrix{Float64}, ::Vector{Float64}, ::Bool). But since that call was already precompiled, if inference ends up good enough that it realized it would call that instead, then bingo compile times dropped from 16 seconds to sub 3. That's a nice showcase that the easiest way to fix compile time issues may be to fix inference issues in a package, because that can make Julia narrow down what methods it needs to compile more, and then it may have a better chance of hitting the pieces that you told it to precompile (via a using-time example in a precompile.jl).

However, even if you have dynamism, you can still improve inference. The DiffEq problem was that ForwardDiff has a dynamically chosen chunksize based on the size of the input ODEs u. We previously would defer this calculation of the chunksize until the caches for the Jacobian were built, so then all dual number caches would have a where N on the chunk size from inference. But a function barrier makes the total runtime cost 100ns, so whatever that's solved right? But because all of the caches have this non-inference of the chunksize, Julia's compiler could not determine all of the N's were the same, so inference would see far too many types in expressions and just give up. That's how A::Matrix{Float64} became Any in the inference passes. When Julia hits the function barrier, it will cause inference to trigger again, in which case inside the function it will now be type-stable, so again no runtime cost, but at the first compile it will compile all possible methods it may ever need to call... which is a much bigger universe than what we actually hit. We noticed this was the case because if the user manually put in a chunksize, i.e. Rodas5(chunk_size = 3), then this all went away. So what we did was setup a hook so that the moment solve is called, it goes "have you defined the chunk size? If not, let's figure it out right away and then hit __solve. By doing this function barrier earlier, it at least knows that the chunk size should always match the N type parameter of the solver algorithm, so all of the Ns are the same, which makes it realize there's less types and use more of the compile caches before.

That one is harder than the others, but it makes the more general point that caching compilation is only good if inference is good enough.

And of course, check out Tim Holy's workshop and JuliaCon video which is immensely helpful at getting started.

https://www.youtube.com/watch?v=rVBgrWYKLHY https://www.youtube.com/watch?v=wXRMwJdEjX4

What are the next steps?

Well, the next steps are to get the compile times below 0.1 seconds everywhere of course. I've already identified some issues to solve:

[x] https://github.com/SciML/DifferentialEquations.jl/issues/785
[ ] https://github.com/SciML/DiffEqFlux.jl/pull/604
[ ] https://github.com/SciML/DiffEqSensitivity.jl/pull/471
[ ] How can we take the non-stiff ODE solvers even lower? There's pieces in the profile like Base.Logging that could probably get added to the Julia system image, etc. This needs a full investigation by someone who really knows their stuff.
[ ] Can we get effective CI for finding compile time regressions?

Are there more things just missing some simple precompiles? Are there some major threads we're missing? Help us identify where all of this is so we can solve it in full.

Compile times, here we come!

Aug 12 '21 23:08 ChrisRackauckas

Posted to Discourse as well: https://discourse.julialang.org/t/22-seconds-to-3-and-now-more-lets-fix-all-of-the-differentialequations-jl-universe-compile-times/66313

Aug 12 '21 23:08 ChrisRackauckas

https://github.com/JuliaLang/julia/pull/41878
https://github.com/JuliaLang/julia/pull/41877
https://github.com/JuliaIO/Tar.jl/pull/123
https://github.com/JuliaLang/Downloads.jl/pull/138
https://github.com/JuliaLang/Pkg.jl/pull/2695
https://github.com/JuliaSIMD/LoopVectorization.jl/pull/316

@timholy I think gathering them all here will make it easier to package up in the end.

Aug 13 '21 12:08 ChrisRackauckas

https://github.com/FluxML/MacroTools.jl/pull/166
https://github.com/JuliaLang/julia/pull/41885
https://github.com/JuliaLang/Pkg.jl/pull/2699
https://github.com/JuliaAlgebra/MultivariatePolynomials.jl/pull/172
https://github.com/FluxML/MacroTools.jl/pull/167
https://github.com/JuliaPlots/RecipesBase.jl/pull/82
https://github.com/JuliaArrays/StaticArrays.jl/pull/950
https://github.com/JuliaLang/julia/pull/41917
https://github.com/JuliaSIMD/CPUSummary.jl/issues/3 (originally the CPUSummary issue identified in https://github.com/JuliaLang/julia/pull/41913)

Aug 13 '21 15:08 timholy

Would moving dependencies over to say ExprTools.jl be helpful? I've known since https://github.com/SciML/MuladdMacro.jl/issues/9 that MacroTools.jl can have some pretty major performance issues, so I've been curious if it's worth the engineering effort to move away from it at a larger scale. @oxinabox ?

Aug 13 '21 15:08 ChrisRackauckas

Would moving dependencies over to say ExprTools.jl be helpful?

I don't really know, I'm not a big user of such packages (I tend to "roll my own" when I need stuff like that).

I think I pretty much have the invalidations fixed wrt to OrdinaryDiffEq (i.e., the ones you didn't take care of on your own), so I am starting to look at DifferentialEquations. The good news is that there appear to be only two substantive problems:

Unitful: this promote_typejoin method triggers ~1500 invalidations (mostly via CSTParser). That's a bit of a dicey method but I don't think they're breaking any rules. It would be nice to come up with an alternative, though, because promote_typejoin is very close to the typesystem and it's quite nice to leave it a singleton method if possible. It only enables one test to pass, and the test fails if you change it slightly, so I think we could make a case for removing it (especially if we can give them something that works better). This method also bothers me because it doesn't actually construct a Quantity. Despite these issues, Unitful is a sophisticated, high-quality, well-developed package but it pushes the boundaries of the type system (it kinda has to). Nevertheless we're seeing some consequences from a few of its edgier choices.
MultivariatePolynomials: these are trouble. The first one is the same near-piracy that we eliminated from LV. The second looks harder. The package has a lot of ambiguities so fixing things is a bit challenging. UPDATE: I may have fixed it, see https://github.com/JuliaAlgebra/MultivariatePolynomials.jl/pull/172.

Aug 13 '21 21:08 timholy

Would moving dependencies over to say ExprTools.jl be helpful?

ExprTools should be preferred for things that it can do. Because it is more correct. (There are weird edge-cases MacroTools gets wrong) But the only thing it can do is splitdef/combinedef. It also has no dependencies.

For prewalk and post-walk those are super easy to define. I think the code that uses prewalk/postwalk is often clearer if it is written directly recursively than if it uses them anyway.

For pattern matching on Expr, maybe use MLStyle.jl? I have heard it is much faster.

Aug 13 '21 22:08 oxinabox

Tada!

julia> using DifferentialEquations, SnoopCompile

julia> function lorenz(du,u,p,t)
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
       end
lorenz (generic function with 1 method)

julia> u0 = [1.0;0.0;0.0]
3-element Vector{Float64}:
 1.0
 0.0
 0.0

julia> tspan = (0.0,100.0)
(0.0, 100.0)

julia> prob = ODEProblem(lorenz,u0,tspan)
ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
timespan: (0.0, 100.0)
u0: 3-element Vector{Float64}:
 1.0
 0.0
 0.0

julia> alg = Tsit5()
Tsit5()

julia> tinf = @snoopi_deep solve(prob,alg)
InferenceTimingNode: 1.061332/1.436110 on Core.Compiler.Timings.ROOT() with 1 direct children

It involves a lot of custom stuff:

(@v1.8) pkg> st
Status `~/.julia/environments/v1.8/Project.toml`
 [1520ce14] AbstractTrees v0.3.4
 [6e4b80f9] BenchmarkTools v1.1.3
 [2a0fbf3d] CPUSummary v0.1.2 `~/.julia/dev/CPUSummary`
 [00ebfdb7] CSTParser v3.2.4
 [d360d2e6] ChainRulesCore v1.3.0 `~/.julia/dev/ChainRulesCore`
 [da1fd8a2] CodeTracking v1.0.6
 [3da002f7] ColorTypes v0.10.12
 [5ae59095] Colors v0.12.8
 [a2441757] Coverage v1.3.2
 [f68482b8] Cthulhu v2.2.0 `~/.julia/dev/Cthulhu`
 [31a5f54b] Debugger v0.6.8
 [2b5f629d] DiffEqBase v6.72.0 `~/.julia/dev/DiffEqBase`
 [b552c78f] DiffRules v1.3.0 `~/.julia/dev/DiffRules`
 [0c46a032] DifferentialEquations v6.19.0 `~/.julia/dev/DifferentialEquations`
 [e30172f5] Documenter v0.27.5
 [7876af07] Example v0.5.3
 [f6369f11] ForwardDiff v0.10.19
 [86fae568] ImageView v0.10.13
 [a98d9a8b] Interpolations v0.13.4
 [92d709cd] IrrationalConstants v0.1.0
 [c3a54625] JET v0.4.6
 [033835bb] JLD2 v0.4.13
 [aa1ae85d] JuliaInterpreter v0.8.20
 [89398ba2] LocalRegistry v0.3.2
 [bdcacae8] LoopVectorization v0.12.64 `~/.julia/dev/LoopVectorization`
 [6f1432cf] LoweredCodeUtils v2.1.2 `~/.julia/dev/LoweredCodeUtils`
 [1914dd2f] MacroTools v0.5.7 `~/.julia/dev/MacroTools`
 [dbb5928d] MappedArrays v0.4.1
 [85b6ec6f] MethodAnalysis v0.4.4
 [102ac46a] MultivariatePolynomials v0.3.18 `~/.julia/dev/MultivariatePolynomials`
 [d8a4904e] MutableArithmetics v0.2.20
 [6fe1bfb0] OffsetArrays v1.10.5
 [1dea7af3] OrdinaryDiffEq v5.61.1 `~/.julia/dev/OrdinaryDiffEq`
 [14b8a8f1] PkgTemplates v0.7.17
 [05854315] PrecompileCacheUtils v0.1.0 `../../dev/PrecompileCacheUtils`
 [c46f51b8] ProfileView v0.6.10
 [92933f4c] ProgressMeter v1.7.1
 [d330b81b] PyPlot v2.9.0
 [3cdcf5f2] RecipesBase v1.1.1 `~/.julia/dev/RecipesBase`
 [f2c3362d] RecursiveFactorization v0.2.2 `~/.julia/dev/RecursiveFactorization`
 [0f4576a4] RegistryCompatTools v0.1.0 `~/.julia/dev/RegistryCompatTools`
 [295af30f] Revise v3.1.19 `~/.julia/dev/Revise`
 [0bca4576] SciMLBase v1.18.4 `~/.julia/dev/SciMLBase`
 [aa65fe97] SnoopCompile v2.7.1 `~/.julia/dev/SnoopCompile`
 [e2b509da] SnoopCompileCore v2.7.1 `~/.julia/dev/SnoopCompile/SnoopCompileCore`
 [47a9eef4] SparseDiffTools v1.16.4 `~/.julia/dev/SparseDiffTools`
 [aedffcd0] Static v0.3.0 `~/.julia/dev/Static`
 [90137ffa] StaticArrays v1.2.12 `~/.julia/dev/StaticArrays`
 [4c63d2b9] StatsFuns v0.9.9
 [d5829a12] TriangularSolve v0.1.3 `~/.julia/dev/TriangularSolve`
 [1986cc42] Unitful v1.9.0 `~/.julia/dev/Unitful`

This is a bespoke build of Julia 1.8 (with several of my pending PRs) and tweaks to quite a few packages, including deleting the problematic Unitful method mentioned in https://github.com/SciML/DifferentialEquations.jl/issues/786#issuecomment-898730215. I'll see about trying to get the rest of this out there and merged to master,

Interestingly, this does not solve the CPUSummary issue, and yet I'm still getting great compile times. So we can afford to keep looking at this issue. CC @chriselrod.

Aug 18 '21 17:08 timholy

well, well, 👏 👏 I'm impressed haha.

Aug 18 '21 17:08 ChrisRackauckas

I can get this using the released versions of packages too, with the exceptions listed in https://github.com/SciML/DifferentialEquations.jl/issues/786#issuecomment-898521817 and the change to Unitful. I think the really big ones are (1) the reduce_empty fallback deletion, (2) the MultivariatePolynomials convert narrowing, and (3) the not-yet pushed Unitful change. (EDIT: hah, even the Unitful change isn't essential.) Plus much of the merged stuff to Julia 1.8.

We'll want to test 1.7 soon and see if you're still able to get good compile times, and if not possibly backport the essential PRs.

Aug 18 '21 18:08 timholy

https://github.com/SciML/ModelingToolkit.jl/pull/1215 highlights a parallel issue that once the symbolic tools get involved, there's a whole extra can of worms to consider. But that might have to wait until a later day.

Aug 18 '21 20:08 ChrisRackauckas

https://github.com/timholy/SnoopCompile.jl/pull/262

Aug 20 '21 08:08 ChrisRackauckas

Does inference/compilation times reported here include the time it takes to run using XXX?

Oct 05 '21 01:10 ma-sadeghi

No, not the using time, but the using time is relatively unchanged.

Dec 05 '21 14:12 ChrisRackauckas

With:

https://github.com/SciML/OrdinaryDiffEq.jl/pull/1535 https://github.com/SciML/OrdinaryDiffEq.jl/pull/1521 https://github.com/JuliaDiff/SparseDiffTools.jl/pull/168 https://github.com/JuliaDiff/SparseDiffTools.jl/pull/167

Passing a chunk size is now down to 2 inference triggers total on static array code, both of which are a bit confusing.

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5(chunk_size = Val{3}())
tinf = @snoopi_deep solve(prob,alg)
solve(prob,alg)

julia> itrigs = inference_triggers(tinf)
2-element Vector{InferenceTrigger}:
 Inference triggered to call setindex!(::Vector{Tuple{Float64, Float64, Float64}}, ::Tuple{Bool, Bool, Bool}, ::Int64) from generate_chunked_partials (C:\Users\accou\.julia\dev\SparseDiffTools\src\differentiation\compute_jacobian_ad.jl:75) with specialization SparseDiffTools.generate_chunked_partials(::Vector{Float64}, ::UnitRange{Int64}, ::Val{3})
 Inference triggered to call OrdinaryDiffEq.jacobian2W!(::Matrix{Float64}, ::LinearAlgebra.UniformScaling{Bool}, ::Float64, ::Matrix{Float64}, ::Bool) called from toplevel

That's from what used to be 46 inference spots, now there should just be 0 or 1 depending on if the chunk size is passed.

Could constant prop fix chunk size inference?

Dec 08 '21 18:12 ChrisRackauckas

https://github.com/JuliaLang/julia/pull/43370 is a Julia Base PR that fixes https://github.com/SciML/DifferentialEquations.jl/issues/785 and halves the FTTP

Dec 08 '21 18:12 ChrisRackauckas

https://github.com/SciML/DifferentialEquations.jl/issues/785 was solved by removing an invalidation in Symbolics: https://github.com/JuliaSymbolics/Symbolics.jl/pull/471 . 🎉 Now using DifferentialEquations (or anything with Symbolics) doesn't invalidate the compile time fixes.

Dec 10 '21 21:12 ChrisRackauckas

ForwardDiff compile time test problem:

using OrdinaryDiffEq, SnoopCompile, ForwardDiff

function lotka_volterra!(du, u, p, t)
  x, y = u
  α, β, δ, γ = p
  du[1] = dx = α*x - β*x*y
  du[2] = dy = -δ*y + γ*x*y
end

# Initial condition
u0 = [1.0, 1.0]

# Simulation interval and intermediary points
tspan = (0.0, 10.0)
tsteps = 0.0:0.1:10.0

# LV equation parameter. p = [α, β, δ, γ]
p = [1.5, 1.0, 3.0, 1.0]

# Setup the ODE problem, then solve
prob = ODEProblem(lotka_volterra!, u0, tspan, p)
sol = solve(prob, Tsit5())

function loss(p)
  sol = solve(prob, Tsit5(), p=p, saveat = tsteps)
  sum(abs2, sol.-1)
end

tinf = @snoopi_deep ForwardDiff.gradient(loss,p)

using ProfileView
ProfileView.view(flamegraph(tinf))

Screenshot 2021-12-10 170305

A lot of this is from ForwardDiff not caching some compiles. @chriselrod let's take a look at that.

Dec 10 '21 22:12 ChrisRackauckas

A lot of this is from ForwardDiff not caching some compiles. @chriselrod let's take a look at that.

As discussed on slack, tagging is likely part of the problem.

A solution I'd suggest is defining a method for some entry point functions like solve to convert from the Dual input type to custom tags defined inside DiffEq. The trick of course would be to avoid allocating and avoid reinterpret for Array inputs, which would still carry the old type. My suggestion there would be to GC.@preserve, cast the pointer, and then use an array type wrapping the pointer.

Basically, add a solve method for Dual{T,V,N}s that automatically casts to Dual{DiffEqTagX,V,N} (and of course extend this to nested duals). The trickiest part of this is probably handling the return values correctly. Probably best to allocate them in the correct return type, then GC.@preserve and cast for all the internal code filling it.

Perhaps it is best to only support StaticArrays initially, so that you don't have to worry about these conversions.

Dec 12 '21 01:12 chriselrod

Basically, add a solve method for Dual{T,V,N}s that automatically casts to Dual{DiffEqTagX,V,N} (and of course extend this to nested duals).

That would go in the high level solve.jl in DiffEqBase.jl

For the internal duals, let's make an option in algorithms to use standardtag=true by default, and then just use DiffEqTag internally.

Dec 12 '21 12:12 ChrisRackauckas

Forward mode AD is a lot faster now:

https://github.com/SciML/DiffEqBase.jl/pull/722 https://github.com/SciML/OrdinaryDiffEq.jl/pull/1546 https://github.com/SciML/OrdinaryDiffEq.jl/pull/1544 https://github.com/SciML/OrdinaryDiffEq.jl/pull/1540

using OrdinaryDiffEq, SnoopCompile, ForwardDiff

lorenz = (du,u,p,t) -> begin
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]; tspan = (0.0,100.0);
prob = ODEProblem(lorenz,u0,tspan); alg = Rodas5();
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)

Before:

#First
InferenceTimingNode: 1.849625/14.538148 on Core.Compiler.Timings.ROOT() with 32 direct children

#Second
InferenceTimingNode: 1.531660/4.170409 on Core.Compiler.Timings.ROOT() with 12 direct children

After:

#First
InferenceTimingNode: 1.181086/3.320321 on Core.Compiler.Timings.ROOT() with 32 direct children

#Second
InferenceTimingNode: 0.998814/1.650488 on Core.Compiler.Timings.ROOT() with 11 direct children

Dec 14 '21 11:12 ChrisRackauckas

@timholy is going to beat all of us in this compile time race by attempting https://github.com/JuliaLang/julia/pull/42016

Dec 14 '21 11:12 ChrisRackauckas

Chunked out some invalidations: https://github.com/SciML/DiffEqBase.jl/pull/723 and the CLIMA folks are happy https://github.com/CliMA/TurbulenceConvection.jl/pull/673.

@timholy is there an easy way to figure out which invalidations you should prioritize?

Dec 23 '21 22:12 ChrisRackauckas

I took some hints from @charleskawczynski and went invalidation hunting:

# From: https://timholy.github.io/SnoopCompile.jl/stable/snoopr/
using SnoopCompile
invalidations = @snoopr begin
    using DifferentialEquations

    function lorenz(du,u,p,t)
     du[1] = 10.0(u[2]-u[1])
     du[2] = u[1]*(28.0-u[3]) - u[2]
     du[3] = u[1]*u[2] - (8/3)*u[3]
    end
    u0 = [1.0;0.0;0.0]
    tspan = (0.0,100.0)
    prob = ODEProblem(lorenz,u0,tspan)
    alg = Rodas5()
    tinf = solve(prob,alg)
end;

trees = SnoopCompile.invalidation_trees(invalidations);

@show length(SnoopCompile.uinvalidated(invalidations)) # show total invalidations

show(trees[end]) # show the most invalidated method

# Count number of children (number of invalidations per invalidated method)
n_invalidations = map(trees) do methinvs
    SnoopCompile.countchildren(methinvs)
end

import Plots
Plots.plot(
    1:length(trees),
    n_invalidations;
    markershape = :circle,
    xlabel = "i-th method invalidation",
    label = "Number of children per method invalidations",
)

Resulting in downstream PRs to remove all of the largest offenders:

https://github.com/JuliaDiff/ChainRulesCore.jl/pull/524 (see https://github.com/JuliaDiff/ChainRulesCore.jl/issues/523)
https://github.com/PainterQubits/Unitful.jl/pull/509

Except for https://github.com/JuliaFolds/InitialValues.jl/issues/63, the last of which I'll want @tkf's help figuring out.

The PRs handle the top 3 at least, so that entire spike should be deleted.

Edit: https://github.com/JuliaFolds/InitialValues.jl/pull/64 joined the fray

Dec 24 '21 01:12 ChrisRackauckas

This is the current compile time benchmarks:

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Tsit5()
tinf = @snoopi_deep solve(prob,alg)
#InferenceTimingNode: 0.753838/1.076605 on Core.Compiler.Timings.ROOT() with 1 direct children

# New Session
alg = Rodas5(chunk_size = Val{3}(),linsolve = DifferentialEquations.OrdinaryDiffEq.RFLUFactorization())
#InferenceTimingNode: 1.069709/1.983534 on Core.Compiler.Timings.ROOT() with 7 direct children

# New Session
using OrdinaryDiffEq, SnoopCompile, ForwardDiff

lorenz = (du,u,p,t) -> begin
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]; tspan = (0.0,100.0);
prob = ODEProblem(lorenz,u0,tspan); alg = Rodas5();
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)

# First: InferenceTimingNode: 1.233956/3.789888 on Core.Compiler.Timings.ROOT() with 35 direct children
# Second: InferenceTimingNode: 1.023062/1.785363 on Core.Compiler.Timings.ROOT() with 15 direct children

Dec 24 '21 01:12 ChrisRackauckas

Now looking at using times, https://github.com/SciML/DifferentialEquations.jl/pull/835 chunked that down. I then started hunting downstream libraries with big Requires:

https://github.com/JuliaSIMD/Polyester.jl/pull/55
https://github.com/JuliaSIMD/LoopVectorization.jl/issues/372

While trying to track down using times I noticed I was missing some tooling. @timholy do you know of a way to make the profiler tell you which package is using Requires.jl time? https://discourse.julialang.org/t/profiling-requires-jl/73570 The profiler says that most of the remaining time is in Requires.jl, but I can't figure out where.

So I commented out the Requires.jl usage in Polyester, LoopVectorization, ArrayInterface, and DiffEqBase. Adding DiffEqBase and ArrayInterface to the list had zero change from just Polyester and LoopVectorization, which means the SciML libraries are not the Requires.jl users that are the trouble, and surprisingly neither is ArrayInterface.jl. I then looked at with and without a custom system image:

using PackageCompiler
create_sysimage(["StaticArrays","ForwardDiff"],sysimage_path="sysimage2.so")

and saw:

# Before PR
julia> @time using DifferentialEquations
  8.694322 seconds (24.77 M allocations: 1.736 GiB, 7.27% gc time, 17.92% compilation time)

# After PR
julia> @time using DifferentialEquations
  5.761738 seconds (18.00 M allocations: 1.327 GiB, 4.54% gc time, 10.38% compilation time)

# After PR + sysimage
julia> @time using DifferentialEquations
  4.565475 seconds (14.80 M allocations: 1.108 GiB, 6.05% gc time, 8.73% compilation time)

That lead to this Discourse thread on what to do about ForwardDiff and StaticArrays.jl contributing about one second of this time. The mysterious other 4.5 seconds in Requires.jl time is still elusive though... I need a Requires profiler!

Dec 24 '21 10:12 ChrisRackauckas

Package load times have improved dramatically by doing ArrayInterfaceCore, and now GPUArraysCore. ArrayInterface changes was about 1 second off everything. Now GPUArrays is another second off. Numbers:

Before:

julia> @time_imports using RecursiveArrayTools 18.3 ms ┌ MacroTools 35.6 ms ┌ ZygoteRules 2.3 ms ┌ Compat 318.7 ms ┌ FillArrays 1320.3 ms ┌ StaticArrays 80.1 ms ┌ Preferences 86.5 ms ┌ JLLWrappers 373.9 ms ┌ LLVMExtra_jll 17.1 ms ┌ CEnum 485.5 ms ┌ LLVM 3.2 ms ┌ Adapt 2731.4 ms ┌ GPUArrays 11.8 ms ┌ DocStringExtensions 2.1 ms ┌ IfElse 72.5 ms ┌ RecipesBase 100.5 ms ┌ Static 6.0 ms ┌ ArrayInterfaceCore 164.4 ms ┌ ArrayInterface 13.0 ms ┌ ArrayInterfaceStaticArrays 208.8 ms ┌ ChainRulesCore 5503.5 ms RecursiveArrayTools

julia> @time_imports using RecursiveArrayTools 28.2 ms ┌ MacroTools 53.8 ms ┌ ZygoteRules 3.2 ms ┌ Compat 503.4 ms ┌ FillArrays 1715.4 ms ┌ StaticArrays 69.1 ms ┌ Preferences 103.9 ms ┌ JLLWrappers 500.3 ms ┌ LLVMExtra_jll 7.0 ms ┌ CEnum 265.1 ms ┌ LLVM 3.1 ms ┌ Adapt 2544.7 ms ┌ GPUArrays 16.1 ms ┌ DocStringExtensions 3.1 ms ┌ IfElse 265.8 ms ┌ RecipesBase 175.6 ms ┌ Static 8.1 ms ┌ ArrayInterfaceCore 222.9 ms ┌ ArrayInterface 3.6 ms ┌ ArrayInterfaceStaticArrays 175.3 ms ┌ ChainRulesCore 6228.5 ms RecursiveArrayTools

After:

julia> @time_imports using RecursiveArrayTools 30.9 ms ┌ MacroTools 54.5 ms ┌ ZygoteRules 2.3 ms ┌ Compat 299.2 ms ┌ FillArrays 1240.2 ms ┌ StaticArrays 7.3 ms ┌ DocStringExtensions 1.5 ms ┌ IfElse 43.5 ms ┌ RecipesBase 149.6 ms ┌ Static 5.2 ms ┌ ArrayInterfaceCore 195.5 ms ┌ ArrayInterface 3.3 ms ┌ Adapt 5.7 ms ┌ ArrayInterfaceStaticArrays 185.8 ms ┌ ChainRulesCore 10.2 ms ┌ GPUArraysCore 2169.1 ms RecursiveArrayTools

julia> @time_imports using RecursiveArrayTools 23.3 ms ┌ MacroTools 44.2 ms ┌ ZygoteRules 2.4 ms ┌ Compat 305.4 ms ┌ FillArrays 1015.0 ms ┌ StaticArrays 12.3 ms ┌ DocStringExtensions 2.0 ms ┌ IfElse 108.6 ms ┌ RecipesBase 134.0 ms ┌ Static 5.8 ms ┌ ArrayInterfaceCore 190.5 ms ┌ ArrayInterface 3.4 ms ┌ Adapt 5.2 ms ┌ ArrayInterfaceStaticArrays 273.1 ms ┌ ChainRulesCore 13.3 ms ┌ GPUArraysCore 2111.9 ms RecursiveArrayTools

@time_imports using OrdinaryDiffEq

Before:

11.2 ms ┌ MacroTools 20.1 ms ┌ ZygoteRules 16.0 ms ┌ Preferences 17.7 ms ┌ JLLWrappers 68.4 ms ┌ Rmath_jll 134.5 ms ┌ Rmath 3.7 ms ┌ Compat 1.5 ms ┌ NaNMath 3.5 ms ┌ Calculus 52.7 ms ┌ ChainRulesCore 55.6 ms ┌ ChangesOfVariables 1.3 ms ┌ OpenLibm_jll 2.3 ms ┌ InverseFunctions 4.8 ms ┌ DocStringExtensions 5.4 ms ┌ IrrationalConstants 1.5 ms ┌ CompilerSupportLibraries_jll 2.0 ms ┌ LogExpFunctions 1.7 ms ┌ OpenSpecFun_jll 29.7 ms ┌ SpecialFunctions 13.7 ms ┌ DualNumbers 121.9 ms ┌ HypergeometricFunctions 1.0 ms ┌ Reexport 281.1 ms ┌ StatsFuns 2.0 ms ┌ CpuId 412.8 ms ┌ StaticArrays 1.1 ms ┌ SIMDTypes 1.2 ms ┌ IfElse 35.6 ms ┌ Static 3.5 ms ┌ ArrayInterfaceCore 56.2 ms ┌ ArrayInterface 1.7 ms ┌ Adapt 45.7 ms ┌ OffsetArrays 48.9 ms ┌ ArrayInterfaceOffsetArrays 3.6 ms ┌ ArrayInterfaceStaticArrays 3.0 ms ┌ ManualMemory 538.4 ms ┌ LayoutPointers 57.3 ms ┌ CPUSummary 1.0 ms ┌ BitTwiddlingConvenienceFunctions 67.9 ms ┌ HostCPUFeatures 1038.9 ms ┌ VectorizationBase 0.9 ms ┌ IteratorInterfaceExtensions 2.6 ms ┌ TableTraits 41.7 ms ┌ ThreadingUtilities 59.1 ms ┌ PolyesterWeave 3.6 ms ┌ DiffResults 13.8 ms ┌ FunctionWrappers 1.8 ms ┌ CommonSubexpressions 2.0 ms ┌ DiffRules 113.9 ms ┌ ForwardDiff 6.1 ms ┌ SLEEFPirates 155.3 ms ┌ SIMDDualNumbers 7.8 ms ┌ OrderedCollections 74.5 ms ┌ DataStructures 1.4 ms ┌ MuladdMacro 13.6 ms ┌ KLU 26.2 ms ┌ LabelledArrays 30.2 ms ┌ PreallocationTools 1.7 ms ┌ Requires 1.2 ms ┌ StatsAPI 6.9 ms ┌ Distances 0.9 ms ┌ UnPack 144.2 ms ┌ FillArrays 1.3 ms ┌ DataValueInterfaces 113.2 ms ┌ LLVMExtra_jll 6.0 ms ┌ CEnum 121.8 ms ┌ LLVM 4.3 ms ┌ GPUArraysCore 825.5 ms ┌ GPUArrays 28.6 ms ┌ RecipesBase 1039.7 ms ┌ RecursiveArrayTools 2.8 ms ┌ DataAPI 1.5 ms ┌ SortingAlgorithms 8.4 ms ┌ Missings 38.7 ms ┌ StatsBase 1.6 ms ┌ ArrayInterfaceGPUArrays 5.0 ms ┌ CloseOpenIntervals 27.6 ms ┌ StrideArraysCore 37.6 ms ┌ Polyester 15.7 ms ┌ Tables 1.2 ms ┌ CommonSolve 2.5 ms ┌ ConstructionBase 1.0 ms ┌ TreeViews 294.0 ms ┌ SciMLBase 74.2 ms ┌ Setfield 348.4 ms ┌ LoopVectorization 14.0 ms ┌ FiniteDiff 89.4 ms ┌ TriangularSolve 22.0 ms ┌ IterativeSolvers 179.2 ms ┌ RecursiveFactorization 1074.1 ms ┌ NonlinearSolve 1.4 ms ┌ libblastrampoline_jll 24.6 ms ┌ GenericSchur 1.7 ms ┌ Parameters 23.7 ms ┌ PDMats 14.2 ms ┌ Krylov 7.9 ms ┌ NLSolversBase 20.0 ms ┌ LineSearches 1.8 ms ┌ Inflate 3.1 ms ┌ DensityInterface 5.6 ms ┌ QuadGK 243.1 ms ┌ Distributions 55.2 ms ┌ KrylovKit 66.9 ms ┌ LinearSolve 1.5 ms ┌ FastClosures 2.6 ms ┌ FastBroadcast 6.2 ms ┌ NLsolve 3.0 ms ┌ SimpleTraits 7.0 ms ┌ ArnoldiMethod 55.6 ms ┌ Graphs 62.5 ms ┌ VertexSafeGraphs 82.7 ms ┌ SparseDiffTools 131.3 ms ┌ DiffEqBase 7.3 ms ┌ ExponentialUtilities 5809.3 ms OrdinaryDiffEq

After:

https://github.com/SciML/RecursiveArrayTools.jl/pull/213 https://github.com/JuliaArrays/ArrayInterface.jl/pull/320 https://github.com/SciML/DiffEqNoiseProcess.jl/pull/106 https://github.com/SciML/LinearSolve.jl/pull/146 https://github.com/SciML/ExponentialUtilities.jl/pull/91

@time_imports using OrdinaryDiffEq

10.4 ms ┌ MacroTools 17.7 ms ┌ ZygoteRules 15.6 ms ┌ Preferences 17.3 ms ┌ JLLWrappers 163.2 ms ┌ Rmath_jll 252.5 ms ┌ Rmath 3.0 ms ┌ Compat 1.3 ms ┌ NaNMath 2.8 ms ┌ Calculus 49.3 ms ┌ ChainRulesCore 51.3 ms ┌ ChangesOfVariables 1.2 ms ┌ OpenLibm_jll 2.1 ms ┌ InverseFunctions 4.1 ms ┌ DocStringExtensions 4.6 ms ┌ IrrationalConstants 1.3 ms ┌ CompilerSupportLibraries_jll 1.8 ms ┌ LogExpFunctions 1.7 ms ┌ OpenSpecFun_jll 15.0 ms ┌ SpecialFunctions 11.5 ms ┌ DualNumbers 98.0 ms ┌ HypergeometricFunctions 0.9 ms ┌ Reexport 372.3 ms ┌ StatsFuns 1.6 ms ┌ CpuId 413.5 ms ┌ StaticArrays 1.2 ms ┌ SIMDTypes 0.9 ms ┌ IfElse 34.7 ms ┌ Static 2.8 ms ┌ ArrayInterfaceCore 52.9 ms ┌ ArrayInterface 1.7 ms ┌ Adapt 40.4 ms ┌ OffsetArrays 43.5 ms ┌ ArrayInterfaceOffsetArrays 3.1 ms ┌ ArrayInterfaceStaticArrays 3.3 ms ┌ ManualMemory 529.6 ms ┌ LayoutPointers 56.7 ms ┌ CPUSummary 1.0 ms ┌ BitTwiddlingConvenienceFunctions 95.7 ms ┌ HostCPUFeatures 1036.7 ms ┌ VectorizationBase 0.8 ms ┌ IteratorInterfaceExtensions 2.4 ms ┌ TableTraits 41.7 ms ┌ ThreadingUtilities 59.5 ms ┌ PolyesterWeave 3.7 ms ┌ DiffResults 12.7 ms ┌ FunctionWrappers 1.3 ms ┌ CommonSubexpressions 2.0 ms ┌ DiffRules 116.5 ms ┌ ForwardDiff 6.1 ms ┌ SLEEFPirates 157.1 ms ┌ SIMDDualNumbers 7.0 ms ┌ OrderedCollections 71.3 ms ┌ DataStructures 1.3 ms ┌ MuladdMacro 13.7 ms ┌ KLU 25.4 ms ┌ LabelledArrays 28.3 ms ┌ PreallocationTools 1.5 ms ┌ Requires 1.2 ms ┌ StatsAPI 6.5 ms ┌ Distances 1.1 ms ┌ UnPack 141.8 ms ┌ FillArrays 1.2 ms ┌ DataValueInterfaces 24.0 ms ┌ RecipesBase 3.2 ms ┌ GPUArraysCore 60.3 ms ┌ RecursiveArrayTools 2.6 ms ┌ DataAPI 1.7 ms ┌ SortingAlgorithms 8.5 ms ┌ Missings 36.1 ms ┌ StatsBase 1.7 ms ┌ ArrayInterfaceGPUArrays 6.1 ms ┌ CloseOpenIntervals 25.8 ms ┌ StrideArraysCore 36.7 ms ┌ Polyester 15.5 ms ┌ Tables 1.1 ms ┌ CommonSolve 1.8 ms ┌ ConstructionBase 1.0 ms ┌ TreeViews 277.6 ms ┌ SciMLBase 71.2 ms ┌ Setfield 312.5 ms ┌ LoopVectorization 14.2 ms ┌ FiniteDiff 85.5 ms ┌ TriangularSolve 20.6 ms ┌ IterativeSolvers 111.1 ms ┌ RecursiveFactorization 945.4 ms ┌ NonlinearSolve 1.2 ms ┌ libblastrampoline_jll 24.0 ms ┌ GenericSchur 23.2 ms ┌ PDMats 1.4 ms ┌ Parameters 13.4 ms ┌ Krylov 6.4 ms ┌ NLSolversBase 16.7 ms ┌ LineSearches 1.3 ms ┌ Inflate 2.2 ms ┌ DensityInterface 4.0 ms ┌ QuadGK 297.9 ms ┌ Distributions 53.6 ms ┌ KrylovKit 63.2 ms ┌ LinearSolve 1.3 ms ┌ FastClosures 2.3 ms ┌ FastBroadcast 4.9 ms ┌ NLsolve 2.5 ms ┌ SimpleTraits 5.4 ms ┌ ArnoldiMethod 48.2 ms ┌ Graphs 53.2 ms ┌ VertexSafeGraphs 71.7 ms ┌ SparseDiffTools 59.6 ms ┌ DiffEqBase 136.5 ms ┌ ExponentialUtilities 4685.2 ms OrdinaryDiffEq

Jun 22 '22 09:06 ChrisRackauckas

Current invalidations report:

using SnoopCompile
invalidations = @snoopr begin
    using OrdinaryDiffEq

    function lorenz(du, u, p, t)
        du[1] = 10.0(u[2] - u[1])
        du[2] = u[1] * (28.0 - u[3]) - u[2]
        du[3] = u[1] * u[2] - (8 / 3) * u[3]
    end
    u0 = [1.0; 0.0; 0.0]
    tspan = (0.0, 100.0)
    prob = ODEProblem{true,false}(lorenz, u0, tspan)
    alg = Rodas5()
    tinf = solve(prob, alg)
end;

trees = SnoopCompile.invalidation_trees(invalidations);

@show length(SnoopCompile.uinvalidated(invalidations)) # show total invalidations

show(trees[end]) # show the most invalidated method

# Count number of children (number of invalidations per invalidated method)
n_invalidations = map(trees) do methinvs
    SnoopCompile.countchildren(methinvs)
end

import Plots
Plots.plot(
    1:length(trees),
    n_invalidations;
    markershape=:circle,
    xlabel="i-th method invalidation",
    label="Number of children per method invalidations"
)

julia> show(trees[end])
inserting convert(::Type{DAEFunction}, f) in SciMLBase at C:\Users\accou\.julia\packages\SciMLBase\3fOCs\src\scimlfunctions.jl:3227 invalidated:
   mt_backedges:  1: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for error(::String, ::Expr) (0 children)
                  2: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for FileIO.querysym_all(::IOStream) (0 children)
                  3: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  4: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  5: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for SnoopCompile._show(::IOContext{IOBuffer}, ::SnoopCompileCore.InferenceTiming) (0 children)
                  6: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.mapreduce_empty_iter(::Function, ::Function, ::Vector{Expr}, ::Base.HasEltype) (0 children)
                  7: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::typeof(identity), ::Function) (1 children)
                  8: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(min)) (2 children)
                  9: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(max)) (6 children)
                 10: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(Base.add_sum)) (6 children)
                 11: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(|)) (22 children)
                 12: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(&)) (24 children)
                 13: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::Function) (1704 children)
                 14: signature convert(::Type{T}, x::T) where T in Base at Base.jl:61 (formerly convert(::Type{T}, x::T) where T in Base at Base.jl:61) triggered MethodInstance for Base.print_to_string(::Symbol, ::String, ::Expr, ::String) (1 children)
                 15: signature convert(::Type{T}, x::T) where T in Base at Base.jl:61 (formerly convert(::Type{T}, x::T) where T in Base at Base.jl:61) triggered MethodInstance for KrylovKit._set_num_threads_warn(::Int64) (1 children)
   backedges: 1: superseding convert(::Type{T}, x::T) where T in Base at Base.jl:61 with MethodInstance for convert(::Type{T}, ::T) where T<:Function (13 children)
   1 mt_cache
false

julia> show(trees[end-1])
inserting !(::Static.False) in Static at C:\Users\accou\.julia\packages\Static\sVI3g\src\Static.jl:427 invalidated:
   mt_backedges:   1: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::AbstractFloat, ::AbstractFloat) (0 children)
                   2: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isbadzero(::typeof(min), ::AbstractFloat) (0 children)
                   3: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                   4: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                   5: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                   6: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                   7: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                   8: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                   9: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  10: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Integer, ::Int64) (0 children)
                  11: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol, Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Nothing, ::Any, ::Symbol, ::Nothing, ::Int64) (0 children)
                  12: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:maxlog,), Tuple{Int64}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  13: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Tuple{ErrorException, Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}}, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Tuple{ErrorException, Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}}}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  14: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::Int64) (0 children)
                  15: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97"{typeof(iszero)})(::Any) (0 children)
                  16: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{<:Base.Order.ReverseOrdering{Base.Order.ForwardOrdering}, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}) (0 children)
                  17: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{<:Base.Order.ForwardOrdering, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}) (0 children)
                  18: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Float64}} where _A<:Base.Order.Ordering) (0 children)
                  19: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Int64) (0 children)
                  20: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Base.ExceptionStack, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Base.ExceptionStack}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  21: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, ErrorException, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{ErrorException}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  22: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (0 children)
                  23: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                  24: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                  25: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                  26: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  27: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  28: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                  29: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  30: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Base.ExceptionStack, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Base.ExceptionStack}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  31: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, ErrorException, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{ErrorException}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  32: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol, Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Nothing, ::Any, ::Symbol, ::Nothing, ::Int64) (0 children)
                  33: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::Char, ::Any) (0 children)
                  34: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.getEntry(::Dict{Char, Any}, ::String) (0 children)
                  35: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.LazilyInitializedFields.lazy_struct(::Expr) (0 children)
                  36: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float64}) (0 children)
                  37: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::Function) (0 children)
                  38: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::LinearAlgebra.var"#251#252") (0 children)
                  39: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{String}) (0 children)
                  40: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) (0 children)  
                  41: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Unsigned, ::Int64) (0 children)
                  42: signature Tuple{typeof(!), Any} triggered MethodInstance for showerror(::IOContext{Base.TTY}, ::MethodError) (0 children)
                  43: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::String) (0 children)
                  44: signature Tuple{typeof(!), Any} triggered MethodInstance for Tar.var"#read_tarball#47"(::Vector{UInt8}, ::Base.DevNull, ::typeof(Tar.read_tarball), ::Function, ::Function, ::Base.Process) (0 children)
                  45: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.Registry.var"#11#14"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd) (0 children)
                  46: signature Tuple{typeof(!), Any} triggered MethodInstance for Tar.var"#read_tarball#47"(::Vector{UInt8}, ::Base.DevNull, ::typeof(Tar.read_tarball), ::Tar.var"#26#28"{Vector{UInt8}, Bool, Bool, Base.Process, String}, ::Function, ::Base.Process) (0 children)
                  47: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.PlatformEngines.var"#26#28"{String}, ::Cmd) (0 children)
                  48: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{Pkg.Types.PackageSpec}) (0 children)
                  49: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Any) (0 children)
                  50: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Tuple{Float64, Float64}}} where _A<:Base.Order.Ordering) (0 children)
                  51: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::Any, ::Any) (0 children)
                  52: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::Base.UUID, ::Any) (0 children)
                  53: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::Base.DevNull) (0 children)     
                  54: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::IOBuffer) (0 children)
                  55: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}, ::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}) (0 children)
                  56: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Union{Missing, Float32}}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  57: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Union{Missing, Float64}}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  58: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Missing}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  59: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Float32}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  60: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Float64}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  61: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::typeof(identity), ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::AbstractVector) (0 children)
                  62: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::typeof(identity), ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::Vector) (0 children)
                  63: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#153#184"{Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  64: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#156#187"{Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  65: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::String) (0 children)
                  66: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#159#190"{Dict{String, Dict{String, Set{String}}}, Set{String}, Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  67: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::Base.PkgId) (0 children)
                  68: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, _A, String} where _A, ::Cmd) (0 children)
                  69: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, Tar.var"#1#2", String}, ::Cmd) (0 children)
                  70: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::UInt8) (0 children)
                  71: signature Tuple{typeof(!), Any} triggered MethodInstance for ==(::Dict{String, Any}, ::Dict{String, Any}) (0 children)
                  72: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{Any}) (0 children)
                  73: signature Tuple{typeof(!), Any} triggered MethodInstance for Base._show_nonempty(::IOContext{Base.TTY}, ::AbstractMatrix, ::String, ::Bool, ::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}) (0 children)
                  74: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Int64}} where _A<:Base.Order.Ordering) (0 children)
                  75: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Base.StackTraces.StackFrame}} where _A<:Base.Order.Ordering) (0 children)
                  76: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.REPLMode._completions(::String, ::Bool, ::Int64, ::Int64) (0 children)
                  77: signature Tuple{typeof(!), Any} triggered MethodInstance for allunique(::AbstractRange) (0 children)
                  78: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Char) (0 children)
                  79: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::GlobalRef, ::Any) (0 children)
                  80: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.do_test_throws(::Test.ExecutionResult, ::Any, ::Any) (0 children)   
                  81: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.eval_test(::Expr, ::Expr, ::LineNumberNode, ::Bool) (0 children)    
                  82: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float16}) (0 children)
                  83: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float32}) (0 children)
                  84: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.on_pkg_load(::Base.PkgId) (0 children)
                  85: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess(::Vector, ::Base.Broadcast.Broadcasted{Nothing, Tuple{Base.OneTo{Int64}}, _A, Tuple{Vector{Any}}} where _A) (0 children)
                  86: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  87: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess_args(::Vector, ::Tuple{Vector{Core.LineInfoNode}}) (0 children)
                  88: signature Tuple{typeof(!), Any} triggered MethodInstance for FileIO.__init__() (0 children)
                  89: signature Tuple{typeof(!), Any} triggered MethodInstance for FlameGraphs.var"#flamegraph!#8"(::Bool, ::Vector{Tuple{Symbol, Symbol}}, ::Int64, ::typeof(FlameGraphs.flamegraph!), ::LeftChildRightSiblingTrees.Node{FlameGraphs.NodeData}, ::Profile.StackFrameTree{Base.StackTraces.StackFrame}) (0 children)
                  90: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess(::Vector, ::Base.Broadcast.Broadcasted{Nothing, Tuple{Base.OneTo{Int64}}, _A, Tuple{Vector{Any}}} where _A) (0 children)
                  91: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess_args(::Vector, ::Tuple{Vector{Core.LineInfoNode}}) (0 children)
                  92: signature Tuple{typeof(!), Any} triggered MethodInstance for JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  93: signature Tuple{typeof(!), Any} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  94: signature Tuple{typeof(!), Any} triggered MethodInstance for SnoopCompile.__init__() (0 children)
                  95: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sortperm#12"(::Base.Sort.QuickSortAlg, ::Function, ::Function, ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sortperm), ::Vector{Float64}) (0 children)
                  96: signature Tuple{typeof(!), Any} triggered MethodInstance for StaticArrays.var"#s25#139"(::Any, ::Any, ::Any, ::Any) (0 children)      
                  97: signature Tuple{typeof(!), Any} triggered MethodInstance for FiniteDiff.__init__() (0 children)
                  98: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.var"#artifact_meta#12"(::Base.BinaryPlatforms.Platform, ::typeof(Artifacts.artifact_meta), ::String, ::Dict{String, Any}, ::String) (0 children)
                  99: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Int64, ::ForwardDiff.Dual{Ty}) where Ty (0 children)
                 100: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.match(::Expr, ::Symbol, ::Dict{Any, Any}) (0 children)        
                 101: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{Any, Vararg{Any}}) (1 children)
                 102: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{}) (1 children)
                 103: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair) (1 children)
                 104: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Any, ::Any, ::Expr) (1 children)
                 105: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Expr, ::Any, ::Expr) (1 children)
                 106: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::String, ::Any) (1 children)
                 107: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                 108: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEditREPL(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (1 children)
                 109: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.REPLMode.var"#command_is_focused#53"{Bool, Int64})() (1 children) 
                 110: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.add_code_to_repl_history(::String) (1 children)
                 111: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (1 children)
                 112: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Int64, ::Any) (2 children)
                 113: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dict{Char, Any}, ::Any) (2 children)
                 114: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Function}, ::Any, ::Any) (2 children)
                 115: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{Dict}}, ::Vector, ::Any) (2 children)    
                 116: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{String}}, ::Vector{String}, ::Any) (2 children)
                 117: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Base.UUID}, ::Base.UUID, ::Any) (2 children)
                 118: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Pkg.Types.Compat}, ::Any, ::Any) (2 children)   
                 119: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL._trimdocs(::Markdown.MD, ::Bool) (2 children)
                 120: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Function, ::Cmd) (3 children)
                 121: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Vector{String}, ::Any) (3 children)
                 122: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dict{String, String}, ::Any) (3 children)
                 123: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Nothing, ::Any) (3 children)
                 124: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Any) (4 children)  
                 125: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Symbol}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 126: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Core.SimpleVector}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 127: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair{Symbol, Any}) (4 children)
                 128: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Bool, String}}, ::Any, ::Any) (4 children)
                 129: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Nothing, String}}, ::Any, ::Any) (4 children)
                 130: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pkg.Versions.VersionRange}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 131: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Pkg.REPLMode.CommandSpec}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 132: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Any}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 133: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{Pkg.Types.Stage1}}, ::Vector{Pkg.Types.Stage1}, ::Any) (4 children)
                 134: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, String}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 135: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.UUID}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 136: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Vector{Base.UUID}}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 137: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{VersionNumber}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 138: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.BinaryPlatforms.AbstractPlatform}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 139: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Set{Int64}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 140: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Union{Nothing, Pkg.Resolve.ResolveLogEntry}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 141: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Base.UUID}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 142: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Pkg.Types.Compat}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 143: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Tuple{Union{Nothing, Base.UUID}, Union{Nothing, Pkg.Types.PackageSpec}, Union{Nothing, Pkg.Types.PackageSpec}}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 144: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Tuple{Base.UUID, String, String, VersionNumber}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 145: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dates.DateTime, ::Any) (4 children)
                 146: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.BinaryPlatforms.Platform}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 147: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{REPL.REPLCompletions.Completion}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 148: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Expr) (4 children)       
                 149: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Union{Int64, Symbol}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (5 children)
                 150: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.Platform, ::Base.BinaryPlatforms.Platform) (5 children)
                 151: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{_A, Base.Order.ForwardOrdering} where _A) (5 children)
                 152: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::VersionNumber, ::Any) (5 children)
                 153: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Base.SHA1, ::Any) (5 children)
                 154: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Bool, ::Any) (5 children)
                 155: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Nothing}, ::Nothing, ::Any) (6 children)        
                 156: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.getEntry(::Dict{Char, Any}, ::Union{Char, String}) (6 children)
                 157: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Base.UUID}, ::Any, ::Any) (7 children)
                 158: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Int64, ::Any) (7 children)
                 159: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#38#40")(::Core.MethodMatch) (7 children)
                 160: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Any) (7 children)        
                 161: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Pkg.API.var"#3#5", Base.Order.ForwardOrdering}) (8 children)
                 162: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.REPLMode.Command(::Pkg.REPLMode.Statement) (9 children)
                 163: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Any}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Base.var"#874#880", Base.Order.ForwardOrdering}) (10 children)
                 164: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Vector{Any}) (11 children)
                 165: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (12 children)
                 166: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Pkg.Types.var"#30#32", Base.Order.ForwardOrdering}) (12 children)
                 167: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Int64}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (12 children)
                 168: signature Tuple{typeof(!), Any} triggered MethodInstance for show(::IO, ::TypeVar) (13 children)
                 169: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Tuple{Float64, Float64}}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (15 children)
                 170: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::Any, ::Any, ::Any, ::Any, ::Expr) (16 children)    
                 171: signature Tuple{typeof(!), Any} triggered MethodInstance for SnoopCompile.var"#next_julia_frame#65"(::Bool, ::Bool, ::typeof(SnoopCompile.next_julia_frame), ::Any, ::Int64, ::Int64) (17 children)
                 172: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.Platform) (18 children)
                 173: signature Tuple{typeof(!), Any} triggered MethodInstance for isequal(::Vector{Any}, ::Vector{Any}) (21 children)
                 174: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Base.SHA1, String}}, ::Any, ::Any) (22 children)
                 175: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.REPLCompletions.get_value(::Expr, ::Module) (22 children)
                 176: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{Char, Any}, ::Any, ::Any) (23 children)
                 177: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{<:Base.Order.ForwardOrdering, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}, ::Int64, ::Int64) (30 children)
                 178: signature Tuple{typeof(!), Any} triggered MethodInstance for isinf(::AbstractFloat) (30 children)
                 179: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::String, ::Any) (31 children)
                 180: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Base.StackTraces.StackFrame}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (37 children)
                 181: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Float64}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (40 children)
                 182: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Any}, ::Any, ::Any) (42 children)
                 183: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::Union{Char, String}, ::Any) (47 children)
                 184: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{<:Base.Order.ReverseOrdering{Base.Order.ForwardOrdering}, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}, ::Int64, ::Int64) (64 children)
                 185: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{_A, Nothing} where _A, ::Nothing, ::Any) (70 children)  
                 186: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (126 children)
                 187: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Any, ::Any) (254 children)
                 188: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for (::ReverseDiff.SkipOptimize{typeof(!=)})(::Int64, ::ReverseDiff.TrackedReal) (1 children)
                 189: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.var"#load_overrides#1"(::Bool, ::typeof(Artifacts.load_overrides)) (1 children)
                 190: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.remove_tp(::Expr) (1 children)
                 191: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.var"#artifact_meta#12"(::Base.BinaryPlatforms.Platform, ::typeof(Artifacts.artifact_meta), ::String, ::Dict{String, Any}, ::String) (1 children)
                 192: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (1 children)
                 193: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for !=(::Int64, ::ForwardDiff.Dual{Ty}) where Ty (1 children)
                 194: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.combinations(::Vector{Symbol}, ::Int64) (1 children)
                 195: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.remove_tp(::Any) (1 children)
                 196: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Base.CoreLogging.logging_error(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Bool) (1 children)
false

julia> show(trees[end-2])
inserting promote_rule(::Type{R}, ::Type{ForwardDiff.Dual{T, V, N}}) where {R<:Real, T, V, N} in ForwardDiff at C:\Users\accou\.julia\packages\ForwardDiff\pDtsf\src\dual.jl:425 invalidated:
   backedges: 1: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{Int64}, ::Type{S} where S<:Real) (5 children)
              2: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{UInt16}, ::Type) (11 children)
              3: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{UInt8}, ::Type) (149 children)
              4: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{Int64}, ::Type) (372 children)
   19 mt_cache
false

julia> show(trees[end-3])
inserting tail(t::ChainRulesCore.Tangent{<:NamedTuple{<:Any, <:Tuple{}}}) in ChainRulesCore at C:\Users\accou\.julia\packages\ChainRulesCore\ctmSK\src\tangent_types\tangent.jl:110 invalidated:
   mt_backedges:  1: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._cshp(::Int64, ::Tuple{Bool}, ::Tuple{Int64}, ::Any) (0 children)
                  2: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._cshp(::Int64, ::Tuple{Bool}, ::Tuple{Any, Vararg{Any}}, ::Any) (0 children)
                  3: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple, ::Any) (0 children)
                  4: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children)
                  5: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Any, Vararg{Any}}, ::Missing) (0 children)
                  6: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Bool, Vararg{Any}}, ::Bool) (0 children)
                  7: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children)
                  8: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for iterate(::Base.Iterators.Enumerate{Vector{VersionNumber}}, ::Any) (0 children)
                  9: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for VSCodeServer.JuliaInterpreter.optimize!(::Core.CodeInfo, ::Method) (0 children)
                 10: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for JuliaInterpreter.optimize!(::Core.CodeInfo, ::Method) (0 children)
                 11: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                 12: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.tail(::NamedTuple{names}) where names (450 children)
false

julia> show(trees[end-4])
inserting checkindex(::Type{Bool}, inds::OffsetArrays.IdOffsetRange, i::Real) in OffsetArrays at C:\Users\accou\.julia\packages\OffsetArrays\80Lkc\src\axes.jl:276 invalidated:
   mt_backedges:  1: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  2: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  3: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, Type{ColorTypes.RGB{FixedPointNumbers.N0f8}}, Tuple{Vector{ColorTypes.LCHab{Float64}}}}) (0 children)    
                  4: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, Type{ColorTypes.RGB{FixedPointNumbers.N0f8}}, Tuple{Vector{ColorTypes.LCHab{Float64}}}}) (0 children)  
                  5: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  6: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  7: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  8: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  9: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for Base.checkbounds_indices(::Type{Bool}, ::Tuple{Any}, ::Tuple{Vararg{Int64}}) (17 children)
                 10: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for Base.checkbounds_indices(::Type{Bool}, ::Tuple, ::Tuple{Integer}) (109 children)
   backedges: 1: superseding checkindex(::Type{Bool}, inds::AbstractUnitRange, i::Real) in Base at abstractarray.jl:727 with MethodInstance for checkindex(::Type{Bool}, ::AbstractUnitRange, ::Int64) (5 children)
              2: superseding checkindex(::Type{Bool}, inds::AbstractUnitRange, i::Real) in Base at abstractarray.jl:727 with MethodInstance for checkindex(::Type{Bool}, ::AbstractUnitRange, ::Integer) (5 children)

invalidations

Aug 21 '22 10:08 ChrisRackauckas

Base.promote_rule(::Type{R}, ::Type{Dual{T,V,N}}) where {R<:$R,T,V,N} = Dual{T,promote_type(R, V),N}

@timholy any ideas for anything that can be done for that invalidation?

Aug 21 '22 10:08 ChrisRackauckas

https://github.com/SciML/DiffEqBase.jl/pull/736 and https://github.com/SciML/OrdinaryDiffEq.jl/pull/1627 together lead to startup times from 8 seconds to 0.7 without a system image, and down to 0.1 and below with a system image.

Without a system image:

using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.847580 seconds (83.25 k allocations: 3.404 MiB, 99.75% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.701598 seconds (499.23 k allocations: 28.846 MiB, 99.73% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000113 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000147 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 8.587653 seconds (24.77 M allocations: 3.581 GiB, 5.37% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.122847 seconds (3.69 M allocations: 211.491 MiB, 2.45% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000120 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.188 KiB)

With a system image:

using PackageCompiler
create_sysimage(["OrdinaryDiffEq"], sysimage_path="DiffEqSysImage.so")



using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.133316 seconds (4.66 k allocations: 309.428 KiB, 2.40% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.097989 seconds (432.99 k allocations: 26.587 MiB, 99.01% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000105 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 7.651692 seconds (23.73 M allocations: 3.516 GiB, 8.12% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.120606 seconds (2.67 M allocations: 145.677 MiB, 27.06% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000098 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000140 seconds (950 allocations: 45.188 KiB)

Before:

jit_lag

After:

no_jit_lag

More interestingly, it puts us into the paradigm where most ODE solve calls are actually fully precompiled, separately from the f call.

Aug 21 '22 13:08 ChrisRackauckas

The function specialization precompilation system is now fully documented:

https://github.com/SciML/SciMLBase.jl/pull/257

https://scimlbase.sciml.ai/stable/interfaces/Problems/#Specialization-Levels

Sep 20 '22 21:09 ChrisRackauckas

DifferentialEquations.jl
DifferentialEquations.jl copied to clipboard

22 seconds to 3 and now more: Let's fix all of the DifferentialEquations.jl + universe compile times!

Take awhile to precompile with style all of the DiffEq stack, no denial

What were our first strides?

Show me some results

The precompilation results do not always generalize

How can I as a user help?

What are some helpful tricks and knowledge to share?

What are the next steps?

DifferentialEquations.jl DifferentialEquations.jl copied to clipboard

22 seconds to 3 and now more: Let's fix all of the DifferentialEquations.jl + universe compile times!

Take awhile to precompile with style all of the DiffEq stack, no denial

What were our first strides?

Show me some results

The precompilation results do not always generalize

How can I as a user help?

What are some helpful tricks and knowledge to share?

What are the next steps?

DifferentialEquations.jl
DifferentialEquations.jl copied to clipboard