Setup FunctionWrappersWrappers norecompile mode #736

ChrisRackauckas · 2022-03-22T10:08:40Z

Needs:

using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(prob, Rosenbrock23(autodiff=false))
@time sol = solve(prob, Rosenbrock23(chunk_size=1))

2.763588 seconds (10.32 M allocations: 648.718 MiB, 4.92% gc time, 99.89% compilation time)

10.577789 seconds (45.44 M allocations: 2.760 GiB, 4.87% gc time, 99.97% compilation time)

While the types of prob are exactly the same, there is still a significant amount of compile time, even with that exact same time being called in using at OrdinaryDiffEq.

Maybe this needs to be run on master?

ChrisRackauckas · 2022-03-22T13:51:46Z

Hmm, @chriselrod confirmed this on Julia master too, so JuliaLang/julia#43990 isn't what was missing.

@timholy I'm confused 😅 . In OrdinaryDiffEq we have this call during using:

https://github.com/SciML/OrdinaryDiffEq.jl/pull/1627/files#diff-21b6a398b057cb5875ecf986247c5dc81faea12d39a3d4587f013775852fc8c2R241

then we have with this new FunctionWrappersWrappers we get typeof(prob) === typeof(lorenzprob) # true, so the types going into the solve are exactly the same as the ones done at using time, but... we get 99% compilation time anyways. Do you know what could trigger such a thing?

ChrisRackauckas · 2022-03-27T13:08:43Z

Compile times increase 😅 . And that's with forcing precompilation on the FunctionWrapper type. So I dug into why.

The following tests are with:

(@v1.8) pkg> st
Status `C:\Users\accou\.julia\environments\v1.8\Project.toml`
  [2b5f629d] DiffEqBase v6.82.2 `C:\Users\accou\.julia\dev\DiffEqBase`
  [069b7b12] FunctionWrappers v1.1.2
  [1dea7af3] OrdinaryDiffEq v6.7.1 `C:\Users\accou\.julia\dev\OrdinaryDiffEq`
  [0bca4576] SciMLBase v1.28.0 `C:\Users\accou\.julia\dev\SciMLBase`
  [aa65fe97] SnoopCompile v2.9.1

The results are pretty disconcerting. I'm copying from https://github.com/PumasAI/Pumas.jl/pull/2188.

It seems the issue is with FunctionWrappers.jl itself. What clued me into this was of course I was trying to do this with DiffEqBase.jl (#736) but was seeing a similarly bad result. To try and figure out why, I started snooping the compiles. What I found was interesting. Full specialized version:

using OrdinaryDiffEq, SnoopCompile

function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)

prob = ODEProblem(f, u0, tspan)
tinf = @snoopi_deep sol = solve(prob, Tsit5())
itrigs = inference_triggers(tinf)

julia> tinf = @snoopi_deep sol = solve(prob, Tsit5())
InferenceTimingNode: 0.713619/1.176800 on Core.Compiler.Timings.ROOT() with 1 direct children

julia> itrigs = inference_triggers(tinf)
1-element Vector{InferenceTrigger}:
 Inference triggered to call CommonSolve.solve(::ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, typeof(f), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}) from eval (.\boot.jl:368) inlined into VSCodeServer.repleval(::Module, ::Expr, ::String) (c:\Users\accou\.vscode\extensions\julialang.language-julia-1.6.5\scripts\packages\VSCodeServer\src\repl.jl:157)

Beautifully perfect inference, right? Now FunctionWerappersWrappers.

using OrdinaryDiffEq, SnoopCompile

function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan)
tinf = @snoopi_deep sol = solve(prob, Tsit5())
itrigs = inference_triggers(tinf)

How does that look?

julia> tinf = @snoopi_deep sol = solve(prob, Tsit5())
InferenceTimingNode: 0.677493/1.285641 on Core.Compiler.Timings.ROOT() with 18 direct children

julia> itrigs = inference_triggers(tinf)
18-element Vector{InferenceTrigger}:
 Inference triggered to call CommonSolve.solve(::ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}) from eval (.\boot.jl:368) inlined into VSCodeServer.repleval(::Module, ::Expr, ::String) (c:\Users\accou\.vscode\extensions\julialang.language-julia-1.6.5\scripts\packages\VSCodeServer\src\repl.jl:157)
 Inference triggered to call OrdinaryDiffEq.var"#__solve#559"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(SciMLBase.__solve), ::ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}) from __solve (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\solve.jl:4) inlined into CommonSolve.solve(::ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}) (C:\Users\accou\.julia\dev\DiffEqBase\src\solve.jl:68)
 Inference triggered to call Rational{Int64}(::Int64, ::Int64) from Rational (.\rational.jl:39) inlined into OrdinaryDiffEq.var"#__solve#559"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(SciMLBase.__solve), ::ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}) (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\solve.jl:4)
 ⋮
 Inference triggered to call OrdinaryDiffEq.apply_step!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit}) from loopheader! (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\integrators\integrator_utils.jl:11) with specialization OrdinaryDiffEq.loopheader!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64},FFloat64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit})
 Inference triggered to call OrdinaryDiffEq.handle_tstop!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit}) from solve! (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\solve.jl:484) with specialization CommonSolve.solve!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit})
 Inference triggered to call OrdinaryDiffEq._postamble!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit}) from postamble! (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\integrators\integrator_utils.jl:120) inlined into CommonSolve.solve!(::OrdinaryDiffEq.ODEIntegrator{Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, SciMLBase.NullParameters, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, SciMLBase.NullParameters, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, DiffEqBase.DEStats}, ODEFunction{true, FunctionWrappersWrappers.FunctionWrappersWrapper{Tuple{FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{Float64}, Vector{Float64}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Vector{Float64}, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, SciMLBase.NullParameters, Float64}}, FunctionWrappers.FunctionWrapper{Nothing, Tuple{Vector{ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}, Vector{Float64}, SciMLBase.NullParameters, ForwardDiff.Dual{ForwardDiff.Tag{DiffEqBase.OrdinaryDiffEqTag, Float64}, Float64, 1}}}}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.Tsit5ConstantCache{Float64, Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Nothing, CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Int64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit}) (C:\Users\accou\.julia\dev\OrdinaryDiffEq\src\solve.jl:486)

What's interesting from this is not the the compile time is slower, but that the inference essentially was destroyed. We've seen in the past with DiffEq that it cannot effectively use precompilation unless there is good inference, and so this effect is likely inhibiting any real use of precompilation caching.

Digging into why this is the case, I saw that Cthulhu flagged that it was because of the lack of specialization in:

mutable struct FunctionWrapper{Ret,Args<:Tuple}
    ptr::Ptr{Cvoid}
    objptr::Ptr{Cvoid}
    obj
    objT
    function (::Type{FunctionWrapper{Ret,Args}})(obj::objT) where {Ret,Args,objT}
        objref = Base.cconvert(Ref{objT}, obj)
        new{Ret,Args,typeof(objref),typeof(objT)}(gen_fptr(Ret, Args, objT),
                      Base.unsafe_convert(Ref{objT}, objref), objref, objT)
    end
    (::Type{FunctionWrapper{Ret,Args}})(obj::FunctionWrapper{Ret,Args}) where {Ret,Args} = obj
end

But of course, obj and objT are the object and type we're trying to hide! So IMO this seems to be a core issue with FunctionWrappers.jl itself, and that would need to be solved before making much more progress here. Because of that, I'll be opening a compiler team priority issue about this, since otherwise we won't get much precompilation in any higher order function scenario.

timholy · 2022-03-27T20:12:43Z

One potential solution is to mimic how Observables handles this, which at least lets you precompile the function you will be wrapping. That said, you can precompile only those things you know will be used in advance, and a main application of FunctionWrappers is to handle the case where users might supply an arbitrary number of custom functions as an argument to a call.

In the end of the day, there is a choice to be made: if runtime performance depends on specializing the innermost functions for the specific user function, and if that list of user functions is essentially infinite, then you will never eliminate inference & compilation. If it's not necessary to specialize the innermost functions (i.e., more like a C DE solver) then you'd like to be able to avoid specialization. However, you may be able to handle this with @nospecialize.

ChrisRackauckas · 2022-03-27T20:15:10Z

I thought FunctionWrappers.jl would essentially @nospecialize "for" the downstream? I guess my question is, how could we make that happen? Indeed, we want this case to be essentially like the case of having a C code solver.

timholy · 2022-03-27T20:19:42Z

You may have to @nospecialize the function you're wrapping and all of its callees (or at least those without predictable or precompiled types).

For example, if the callee will only take a Float64 you can do this:

f(x::Float64) = ...
precompile(f, (Float64,))

but if it can take ::Real then you should consider

f(@nospecialize(x::Real)) = ...
precompile(f, (Real,))

ChrisRackauckas · 2022-03-27T20:51:26Z

I tried using @nospecialize but it didn't make a dent. See the latest commit.

chriselrod · 2022-03-27T22:37:56Z

You may have to @nospecialize the function you're wrapping and all of its callees (or at least those without predictable or precompiled types).

The point of function wrappers is to basically be a function pointer with known argument and return types.
Seems like it should infer without a problem.
I didn't think obj and objT were ever accesses. I thought they only existed to protect them from the GC, in case they were (for examples) closures holding some other objects.

We definitely want specialization. A lack of it is going to hurt runtime performance too severely, even if they allow compilation to be cached.

I can experiment with defining my own FunctionWrapper that does not have obj or objT fields.

timholy · 2022-03-28T08:46:07Z

As stated in #736 (comment), if the argtype is deterministic then there shouldn't be any cost to specialization, and indeed you want it for the run-time performance benefits.

But you do want to make sure the f that you supply as an arg to void has been specialized for all of those combinations. Can you have void also return a function that forces precompilation of f for all the methods of f2 that it defines? That's how Observables handles this.

ChrisRackauckas · 2022-03-28T09:25:18Z

I'm not sure how to do that in a way for the user. See the latest commits.

@time

This is a continuation of #736 instead using OpaqueClosures. As such, it also requires: - SciML/SciMLBase.jl#143 - SciML/OrdinaryDiffEq.jl#1627 But now the MWE doesn't run: ```julia using OrdinaryDiffEq function f(du, u, p, t) du[1] = 0.2u[1] du[2] = 0.4u[2] end u0 = ones(2) tspan = (0.0, 1.0) prob = ODEProblem{true,false}(f, u0, tspan, Float64[]) function lorenz(du, u, p, t) du[1] = 10.0(u[2] - u[1]) du[2] = u[1] * (28.0 - u[3]) - u[2] du[3] = u[1] * u[2] - (8 / 3) * u[3] end lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[]) typeof(prob) === typeof(lorenzprob) # true @time sol = solve(prob, Rosenbrock23(autodiff=false)) @time sol = solve(prob, Rosenbrock23(chunk_size=1)) ``` ```julia [ Info: Precompiling OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed] fatal: error thrown and no exception handler available. ErrorException("unimplemented: serialization of MethodInstances for OpaqueClosure") jl_error at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:41 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:686 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:526 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:661 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:675 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:834 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_datatype at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:310 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:605 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:357 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:345 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_save_incremental at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:2289 jl_write_compiler_output at /cygdrive/c/buildbot/worker/package_win64/build/src\precompile.c:65 jl_atexit_hook at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:211 jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:702 mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42 BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line) RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line) ``` The precompilation from SciML/OrdinaryDiffEq.jl#1627 seems to not work with opaque closures. Do they not support precompilation? @Keno

chriselrod · 2022-04-26T20:50:52Z

julia> using SnoopCompile

julia> using OrdinaryDiffEq

julia> function f(du, u, p, t)
           du[1] = 0.2u[1]
           du[2] = 0.4u[2]
       end
f (generic function with 1 method)

julia> u0 = ones(2)
tno2-element Vector{Float64}:
 a1.0
 1.0

julia> tspan = (0.0, 1.0)
(0.0, 1.0)

julia> prob = ODEProblem{true,false}(f, u0, tspan, Float64[])
d ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
timespan: (0.0, 1.0)
u0: 2-element Vector{Float64}:
 1.0
 1.0

julia> function lorenz(du, u, p, t)
           du[1] = 10.0(u[2] - u[1])
           du[2] = u[1] * (28.0 - u[3]) - u[2]
           du[3] = u[1] * u[2] - (8 / 3) * u[3]
       end
lorenz (generic function with 1 method)

julia> lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
=ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
timespan: (0.0, 1.0)
u0: 3-element Vector{Float64}:
 1.0
 0.0
 0.0

julia> typeof(prob) === typeof(lorenzprob) # true
true

julia> tnoad = @snoopi_deep solve(prob, Rosenbrock23(autodiff=false));

julia> tadch1 = @snoopi_deep solve(prob, Rosenbrock23(chunk_size=1));

julia> accumulate_by_source(flatten(tadch1))
19-element Vector{Tuple{Float64, Union{Method, Core.MethodInstance}}}:
 (4.6453e-5, prepare_alg(alg, u0, p, f) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:314)
 (5.1258e-5, isempty(x::Tuple) in Base at tuple.jl:497)
 (5.3864e-5, argtail(x, rest...) in Base at essentials.jl:233)
 (9.1787e-5, (Array{T})(::UndefInitializer, d::Tuple{Vararg{Int64, N}}) where {T, N} in Core at boot.jl:476)
 (9.1915e-5, Vector{T}(::UndefInitializer, d::Tuple{Int64}) where T in Core at boot.jl:468)
 (9.6916e-5, getindex(t::Tuple, i::Int64) in Base at tuple.jl:29)
 (0.000135268, Vector{T}(::UndefInitializer, m::Int64) where T in Core at boot.jl:459)
 (0.000169537, NamedTuple{names, T}(args::T) where {names, T<:Tuple} in Core at boot.jl:607)
 (0.000172067, (::DiffEqBase.var"#get_concrete_problem##kw")(::Any, ::typeof(DiffEqBase.get_concrete_problem), prob, isadapt) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:249)
 (0.00039723499999999996, getproperty(x, f::Symbol) in Base at Base.jl:38)
 (0.000559124, solve_up(prob::SciMLBase.DEProblem, sensealg, u0, p, args...; kwargs...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:168)
 (0.000560135, solve_call(_prob, args...; merge_callbacks, kwargs...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:138)
 (0.000597162, (NamedTuple{names})(args::Tuple) where names in Core at boot.jl:603)
 (0.000614868, solve(prob::SciMLBase.DEProblem, args...; sensealg, u0, p, kwargshandle, kwargs...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:157)
 (0.001569457, var"#solve_call#39"(merge_callbacks, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(DiffEqBase.solve_call), _prob, args...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:138)
 (0.0017321379999999998, var"#solve#40"(sensealg, u0, p, kwargshandle, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(solve), prob::SciMLBase.DEProblem, args...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:157)
 (0.004089142, tail(x::Tuple) in Base at essentials.jl:251)
 (0.004112703, var"#solve_up#41"(kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(DiffEqBase.solve_up), prob::SciMLBase.DEProblem, sensealg, u0, p, args...) in DiffEqBase at /home/chriselrod/.julia/dev/DiffEqBase/src/solve.jl:168)
 (3.596999656, ROOT() in Core.Compiler.Timings at compiler/typeinfer.jl:77)

julia> tadch1
InferenceTimingNode: 3.597000/3.612141 on Core.Compiler.Timings.ROOT() with 3 direct children

However, all their reported times are very small compared to ROOT().
Meaning very little time was spent in inference.

If I disable the precompile statement, I get >4s total time with comparable ROOT() time.
So it seems like it's working?

This is on Julia 1.8.0-beta3.

@time

This is a continuation of #736 instead using OpaqueClosures. As such, it also requires: - SciML/SciMLBase.jl#143 - SciML/OrdinaryDiffEq.jl#1627 But now the MWE doesn't run: ```julia using OrdinaryDiffEq function f(du, u, p, t) du[1] = 0.2u[1] du[2] = 0.4u[2] end u0 = ones(2) tspan = (0.0, 1.0) prob = ODEProblem{true,false}(f, u0, tspan, Float64[]) function lorenz(du, u, p, t) du[1] = 10.0(u[2] - u[1]) du[2] = u[1] * (28.0 - u[3]) - u[2] du[3] = u[1] * u[2] - (8 / 3) * u[3] end lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[]) typeof(prob) === typeof(lorenzprob) # true @time sol = solve(prob, Rosenbrock23(autodiff=false)) @time sol = solve(prob, Rosenbrock23(chunk_size=1)) ``` ```julia [ Info: Precompiling OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed] fatal: error thrown and no exception handler available. ErrorException("unimplemented: serialization of MethodInstances for OpaqueClosure") jl_error at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:41 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:686 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:526 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:661 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:675 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:834 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_datatype at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:310 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:605 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:357 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:345 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_save_incremental at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:2289 jl_write_compiler_output at /cygdrive/c/buildbot/worker/package_win64/build/src\precompile.c:65 jl_atexit_hook at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:211 jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:702 mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42 BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line) RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line) ``` The precompilation from SciML/OrdinaryDiffEq.jl#1627 seems to not work with opaque closures. Do they not support precompilation? @Keno

@time

This is a continuation of #736 instead using OpaqueClosures. As such, it also requires: - SciML/SciMLBase.jl#143 - SciML/OrdinaryDiffEq.jl#1627 But now the MWE doesn't run: ```julia using OrdinaryDiffEq function f(du, u, p, t) du[1] = 0.2u[1] du[2] = 0.4u[2] end u0 = ones(2) tspan = (0.0, 1.0) prob = ODEProblem{true,false}(f, u0, tspan, Float64[]) function lorenz(du, u, p, t) du[1] = 10.0(u[2] - u[1]) du[2] = u[1] * (28.0 - u[3]) - u[2] du[3] = u[1] * u[2] - (8 / 3) * u[3] end lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[]) typeof(prob) === typeof(lorenzprob) # true @time sol = solve(prob, Rosenbrock23(autodiff=false)) @time sol = solve(prob, Rosenbrock23(chunk_size=1)) ``` ```julia [ Info: Precompiling OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed] fatal: error thrown and no exception handler available. ErrorException("unimplemented: serialization of MethodInstances for OpaqueClosure") jl_error at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:41 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:686 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:526 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:661 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:675 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:834 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_datatype at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:310 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:605 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:357 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:345 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_save_incremental at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:2289 jl_write_compiler_output at /cygdrive/c/buildbot/worker/package_win64/build/src\precompile.c:65 jl_atexit_hook at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:211 jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:702 mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42 BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line) RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line) ``` The precompilation from SciML/OrdinaryDiffEq.jl#1627 seems to not work with opaque closures. Do they not support precompilation? @Keno

ChrisRackauckas · 2022-08-20T04:14:28Z

With SnoopPrecompile, norecompile mode is now a very substantial compile time decrease!

using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.871556 seconds (508.96 k allocations: 29.197 MiB, 99.70% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(chunk_size=1))
# 0.766246 seconds (70.68 k allocations: 2.929 MiB, 99.65% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000144 seconds (950 allocations: 45.547 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(chunk_size=1))
# 0.000138 seconds (673 allocations: 68.594 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 8.901039 seconds (26.31 M allocations: 3.869 GiB, 5.77% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(chunk_size=1))
# 2.060342 seconds (6.37 M allocations: 542.085 MiB, 3.91% gc time, 99.85% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000122 seconds (950 allocations: 45.188 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(chunk_size=1))
# 0.000122 seconds (673 allocations: 67.812 KiB)

ChrisRackauckas · 2022-08-20T04:16:57Z

using OrdinaryDiffEq, SnoopCompile
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

tadch1 = @snoopi_deep solve(prob, Rosenbrock23(chunk_size=1));

julia> accumulate_by_source(flatten(tadch1))
6-element Vector{Tuple{Float64, Union{Method, Core.MethodInstance}}}:
 (7.75e-5, Vector{T}(::UndefInitializer, d::Tuple{Int64}) where T in Core at boot.jl:468)
 (8.88e-5, (Array{T})(::UndefInitializer, d::Tuple{Vararg{Int64, N}}) where {T, N} in Core at boot.jl:476)
 (0.0001195, Vector{T}(::UndefInitializer, m::Int64) where T in Core at boot.jl:459)
 (0.0001459, NamedTuple{names, T}(args::T) where {names, T<:Tuple} in Core at boot.jl:607)
 (0.0002753, (NamedTuple{names})(args::Tuple) where names in Core at boot.jl:603)
 (1.0345707, ROOT() in Core.Compiler.Timings at compiler/typeinfer.jl:77)

@time

This is a continuation of #736 instead using OpaqueClosures. As such, it also requires: - SciML/SciMLBase.jl#143 - SciML/OrdinaryDiffEq.jl#1627 But now the MWE doesn't run: ```julia using OrdinaryDiffEq function f(du, u, p, t) du[1] = 0.2u[1] du[2] = 0.4u[2] end u0 = ones(2) tspan = (0.0, 1.0) prob = ODEProblem{true,false}(f, u0, tspan, Float64[]) function lorenz(du, u, p, t) du[1] = 10.0(u[2] - u[1]) du[2] = u[1] * (28.0 - u[3]) - u[2] du[3] = u[1] * u[2] - (8 / 3) * u[3] end lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[]) typeof(prob) === typeof(lorenzprob) # true @time sol = solve(prob, Rosenbrock23(autodiff=false)) @time sol = solve(prob, Rosenbrock23(chunk_size=1)) ``` ```julia [ Info: Precompiling OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed] fatal: error thrown and no exception handler available. ErrorException("unimplemented: serialization of MethodInstances for OpaqueClosure") jl_error at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:41 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:686 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:526 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:661 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:675 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:870 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:834 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_datatype at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:310 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:605 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:357 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_module at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:345 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:733 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:565 jl_serialize_value_ at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:510 [inlined] jl_save_incremental at /cygdrive/c/buildbot/worker/package_win64/build/src\dump.c:2289 jl_write_compiler_output at /cygdrive/c/buildbot/worker/package_win64/build/src\precompile.c:65 jl_atexit_hook at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:211 jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:702 mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42 BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line) RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line) ``` The precompilation from SciML/OrdinaryDiffEq.jl#1627 seems to not work with opaque closures. Do they not support precompilation? @Keno

@time

Needs: - SciML/SciMLBase.jl#143 - SciML/OrdinaryDiffEq.jl#1627 ```julia using OrdinaryDiffEq function f(du, u, p, t) du[1] = 0.2u[1] du[2] = 0.4u[2] end u0 = ones(2) tspan = (0.0, 1.0) prob = ODEProblem{true,false}(f, u0, tspan, Float64[]) function lorenz(du, u, p, t) du[1] = 10.0(u[2] - u[1]) du[2] = u[1] * (28.0 - u[3]) - u[2] du[3] = u[1] * u[2] - (8 / 3) * u[3] end lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[]) typeof(prob) === typeof(lorenzprob) # true @time sol = solve(prob, Rosenbrock23(autodiff=false)) @time sol = solve(prob, Rosenbrock23(chunk_size=1)) ``` ``` 2.763588 seconds (10.32 M allocations: 648.718 MiB, 4.92% gc time, 99.89% compilation time) 10.577789 seconds (45.44 M allocations: 2.760 GiB, 4.87% gc time, 99.97% compilation time) ``` While the types of `prob` are exactly the same, there is still a significant amount of compile time, even with that exact same time being called in `using` at OrdinaryDiffEq. Maybe this needs to be run on master?

ChrisRackauckas · 2022-08-21T08:56:51Z

Without a system image:

using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.847580 seconds (83.25 k allocations: 3.404 MiB, 99.75% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.701598 seconds (499.23 k allocations: 28.846 MiB, 99.73% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000113 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000147 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 8.587653 seconds (24.77 M allocations: 3.581 GiB, 5.37% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.122847 seconds (3.69 M allocations: 211.491 MiB, 2.45% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000120 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.188 KiB)

With a system image:

using PackageCompiler
create_sysimage(["OrdinaryDiffEq"], sysimage_path="DiffEqSysImage.so")



using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.133316 seconds (4.66 k allocations: 309.428 KiB, 2.40% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.097989 seconds (432.99 k allocations: 26.587 MiB, 99.01% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000105 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 7.651692 seconds (23.73 M allocations: 3.516 GiB, 8.12% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.120606 seconds (2.67 M allocations: 145.677 MiB, 27.06% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000098 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000140 seconds (950 allocations: 45.188 KiB)

ChrisRackauckas · 2022-08-21T09:47:43Z

Before:

After:

ChrisRackauckas force-pushed the norecompile branch from c23d5bc to f2ed77b Compare March 25, 2022 02:59

ChrisRackauckas mentioned this pull request Mar 27, 2022

Can't get rid of the last inference trigger? SciML/OrdinaryDiffEq.jl#1632

Closed

ChrisRackauckas force-pushed the norecompile branch from d986e72 to dac395a Compare April 22, 2022 11:23

ChrisRackauckas mentioned this pull request Apr 22, 2022

Setup OpaqueClosures norecompile mode #745

Closed

ChrisRackauckas force-pushed the norecompile branch from dac395a to 954ee43 Compare August 20, 2022 03:39

ChrisRackauckas added 9 commits August 20, 2022 17:58

with extra specialization!

46034cb

try @nospecialize on the arguments

c6ed835

force precompilation?

cba1c32

try more

fa78bbf

no precompile file

449c74d

add sys

e468099

do not require SciMLBase changes and support oop

9675733

fix oop handling

c91f55d

ChrisRackauckas force-pushed the norecompile branch from c0e7570 to c91f55d Compare August 20, 2022 21:58

ChrisRackauckas closed this Aug 21, 2022

ChrisRackauckas reopened this Aug 21, 2022

ChrisRackauckas merged commit b857c3c into master Aug 21, 2022

ChrisRackauckas deleted the norecompile branch August 21, 2022 13:49

ChrisRackauckas mentioned this pull request Aug 21, 2022

22 seconds to 3 and now more: Let's fix all of the DifferentialEquations.jl + universe compile times! SciML/DifferentialEquations.jl#786

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Setup FunctionWrappersWrappers norecompile mode #736

Setup FunctionWrappersWrappers norecompile mode #736

Uh oh!

ChrisRackauckas commented Mar 22, 2022

Uh oh!

ChrisRackauckas commented Mar 22, 2022 •

edited

Loading

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

timholy commented Mar 27, 2022 •

edited

Loading

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

timholy commented Mar 27, 2022

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

chriselrod commented Mar 27, 2022 •

edited

Loading

Uh oh!

timholy commented Mar 28, 2022

Uh oh!

ChrisRackauckas commented Mar 28, 2022

Uh oh!

chriselrod commented Apr 26, 2022 •

edited

Loading

Uh oh!

ChrisRackauckas commented Aug 20, 2022

Uh oh!

ChrisRackauckas commented Aug 20, 2022

Uh oh!

ChrisRackauckas commented Aug 21, 2022

Uh oh!

ChrisRackauckas commented Aug 21, 2022

Uh oh!

Uh oh!

Uh oh!

Setup FunctionWrappersWrappers norecompile mode #736

Setup FunctionWrappersWrappers norecompile mode #736

Uh oh!

Conversation

ChrisRackauckas commented Mar 22, 2022

Uh oh!

ChrisRackauckas commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

timholy commented Mar 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

timholy commented Mar 27, 2022

Uh oh!

ChrisRackauckas commented Mar 27, 2022

Uh oh!

chriselrod commented Mar 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timholy commented Mar 28, 2022

Uh oh!

ChrisRackauckas commented Mar 28, 2022

Uh oh!

chriselrod commented Apr 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChrisRackauckas commented Aug 20, 2022

Uh oh!

ChrisRackauckas commented Aug 20, 2022

Uh oh!

ChrisRackauckas commented Aug 21, 2022

Uh oh!

ChrisRackauckas commented Aug 21, 2022

Uh oh!

Uh oh!

ChrisRackauckas commented Mar 22, 2022 •

edited

Loading

timholy commented Mar 27, 2022 •

edited

Loading

chriselrod commented Mar 27, 2022 •

edited

Loading

chriselrod commented Apr 26, 2022 •

edited

Loading