easy ways to cut off some latency

[tiemvanderdeure]

Inspired by a long slack discussion in gripes, I just ran a quick snoop_compile on this package, and found this one maybe problematic method.

Our definition of Base.convert(T::Type{<:AbstractString}, name::AbstractName) = convert(T, string(name)) seems particularly bad: https://github.com/rafaqz/DimensionalData.jl/blob/d9ac254358c6e689a3de3daa2c320a977bfaabfc/src/name.jl#L8C1-L8C87

Do we need that at all? It leads to:

inserting convert(T::Type{<:AbstractString}, name::DimensionalData.AbstractName) @ DimensionalData ~/.julia/dev/DimensionalData/src/name.jl:8 invalidated:
   mt_backedges:  1: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Base.UUID}, ::Any, ::Any) (0 children)
                  2: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Union{Bool, String}}, ::Any, ::Any) (0 children)
                  3: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Union{Nothing, String}}, ::Any, ::Any) (0 children)
                  4: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Nothing}, ::Nothing, ::Any) (0 children)
                  5: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Union{Nothing, Tuple{Base.PkgId, String}}}, ::Any, ::Any) (0 children)
                  6: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for push!(::Vector{String}, ::Any) (0 children)
                  7: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Dict{String, Pkg.Types.Compat}, ::Any, ::Any) (0 children)
                  8: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for (::Base.var"#cvt1#1"{Tuple{Revise.PkgData, String}, <:Tuple{Revise.PkgData, Any}})(::Int64) (0 children)
                  9: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.JSONRPC.Request(::Any, ::Any, ::Any, ::Nothing) (1 children)
                 10: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.JSONRPC.Request(::Any, ::Any, ::Any, ::VSCodeServer.CancellationTokens.CancellationToken) (1 children)
                 11: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.SubTree(::String, ::Any, ::Any, ::VSCodeServer.Location) (1 children)
                 12: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.ReplWorkspaceItem(::String, ::Any, ::Any, ::Any, ::Any, ::Any, ::Bool, ::String, ::VSCodeServer.Location) (1 children)
                 13: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.ReplRunCodeRequestParams(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (1 children)
                 14: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.SubTree(::String, ::Any, ::Any, ::Nothing) (2 children)
                 15: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for VSCodeServer.ReplWorkspaceItem(::String, ::Any, ::Any, ::Any, ::Any, ::Any, ::Bool, ::String, ::Nothing) (2 children)
                 16: signature Tuple{typeof(convert), Union{Type{String}, Type{Revise.PkgData}}, Any} triggered MethodInstance for (::Base.var"#cvt1#1"{Tuple{Revise.PkgData, String}, <:Tuple{Revise.PkgData, Any}})(::Int64) (3 children)
                 17: signature Tuple{typeof(convert), Union{Type{String}, Type{Revise.PkgData}}, Any} triggered MethodInstance for (::Base.var"#cvt1#1"{Tuple{Revise.PkgData, String}, <:Tuple{Any, String}})(::Int64) (3 children)
                 18: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for setindex!(::Vector{String}, ::Any, ::Int64) (7 children)
                 19: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for convert(::Type{Union{Nothing, String}}, ::Any) (1040 children)

Another thing is SparseArrays. I know it is a stdlib - but we don't really use it in the whole package except for some dispatches of copyto! which would otherwise be ambiguous. Maybe we should consider putting it into an extension? @time_imports shows a big portion of using DimensionalData is just SparseArrays. But maybe it's in so many environments that it's not worth the effort.

      1.3 ms  Statistics
               ┌ 1.3 ms SuiteSparse_jll.__init__() 
    173.0 ms  SuiteSparse_jll 94.88% compilation time
      1.9 ms  Serialization
               ┌ 11.3 ms SparseArrays.CHOLMOD.__init__() 96.12% compilation time
    230.6 ms  SparseArrays 4.70% compilation time
      0.4 ms  Statistics → SparseArraysExt
      0.5 ms  Adapt
      1.4 ms  ArrayInterface
      0.4 ms  ArrayInterface → ArrayInterfaceSparseArraysExt
      0.5 ms  ConstructionBase
      0.4 ms  ConstructionBase → ConstructionBaseLinearAlgebraExt
      1.0 ms  DataAPI
      3.4 ms  Extents
      3.0 ms  Interfaces
     15.8 ms  IntervalSets
      0.4 ms  ConstructionBase → ConstructionBaseIntervalSetsExt
      0.6 ms  IntervalSets → IntervalSetsRandomExt
      0.2 ms  IntervalSets → IntervalSetsStatisticsExt
      3.3 ms  InvertedIndices
      0.3 ms  IteratorInterfaceExtensions
     16.3 ms  Preferences
      0.5 ms  PrecompileTools
     21.6 ms  RecipesBase
      0.4 ms  IntervalSets → IntervalSetsRecipesBaseExt
      0.3 ms  TableTraits
      0.2 ms  DataValueInterfaces
     16.6 ms  Tables
    206.9 ms  DimensionalData

[rafaqz]

Yeah convert should just be for String.

And totally SparseArrays.jl code can go in an extension. (There didn't used to be a loading time cost to SparseArrays)

[asinghvi17]

I am clauding this up, PR incoming

[asinghvi17]

#1048 makes sparsearrays a weakdep

[tiemvanderdeure]

another really bad offenders is apparently this merge method

https://github.com/rafaqz/DimensionalData.jl/blob/3a8c36332fb573c72c674bf2bddc5ae8d936d49f/src/stack/stack.jl#L211-L213

I'm getting

 inserting merge(s::AbstractDimStack, pairs; kw...) @ DimensionalData C:\Users\tsh371\.julia\dev\DimensionalData\src\stack\stack.jl:211 invalidated:
   mt_backedges: 1: signature Tuple{typeof(merge), Any, @NamedTuple{delim_flags::UInt16}} triggered MethodInstance for Base.JuliaSyntax.parse_brackets(::Function, ::Base.JuliaSyntax.ParseState, ::Base.JuliaSyntax.Kind, ::Bool) (1155 children)

1155 children seems pretty bad?

[tiemvanderdeure]

A lot of the invalidations are also in src/Lookups/beginend.jl, but much of it is not very easy to read code and I haven't diven deeper into it. But it's kind of wild test runs now take 50+ minutes on 1.12 (macOS seems particularly bad)