default behavior for flatten/unflatten with any type

[FerreolS]

trafficstars

Hi, I'm using ParameterHandling only for the flatten/unflatten tools. In my case I need to carry extra information within my tuple stored in different type that do not need to be flatten but I need to recover it in my restructured NamedTuple Here is an example

struct MyType
       a::Float64
       b::Int
end

mynametuple = (;a=rand(2),b=randn(6),m = MyType(1,2))

If I try to flatten this tuple I get an error:

flatvector, restructure = ParameterHandling.flatten( mynametuple)
ERROR: MethodError: no method matching flatten(::Type{Float64}, ::MyType)

Closest candidates are:
  flatten(::Type{T}, ::Tuple{}) where T<:Real
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:92
  flatten(::Type{T}, ::Nothing) where T<:Real
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:22
  flatten(::Type{T}, ::Integer) where T<:Real
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:28
  ...

Stacktrace:
 [1] flatten(::Type{Float64}, x::Tuple{MyType})
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:82
 [2] flatten(::Type{Float64}, x::Tuple{Vector{Float64}, MyType})
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:83
 [3] flatten(::Type{Float64}, x::Tuple{Vector{Float64}, Vector{Float64}, MyType})
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:83
 [4] flatten(::Type{Float64}, x::@NamedTuple{a::Vector{Float64}, b::Vector{Float64}, m::MyType})
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:99
 [5] flatten(x::@NamedTuple{a::Vector{Float64}, b::Vector{Float64}, m::MyType})
   @ ParameterHandling ~/.julia/packages/ParameterHandling/WkHof/src/flatten.jl:20
 [6] top-level scope
   @ REPL[14]:1

However, with this simple overloading of ParameterHandling.flatten as a default behaviour

function ParameterHandling.flatten(::Type{T}, x) where {T<:Real}
           v = T[]
           unflatten_to_Any(::Vector{T}) = x
           return v, unflatten_to_Any
 end

I get:

julia> flatvector, restructure = ParameterHandling.flatten( mynametuple)
([0.8409925019315632, 0.2819056254106582, 1.598581283953101, 0.3921407396316173, -1.6414335944803775, 1.0503960727007877, -1.183085803897985, -2.3827810941365573], ParameterHandling.var"#unflatten_to_NamedTuple#15"{Float64, @NamedTuple{a::Vector{Float64}, b::Vector{Float64}, m::MyType}, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_empty_Tuple#14"{Float64, Tuple{}}, var"#unflatten_to_Any#6"{Float64, MyType}}, ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}}, ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}}}((a = [0.8409925019315632, 0.2819056254106582], b = [1.598581283953101, 0.3921407396316173, -1.6414335944803775, 1.0503960727007877, -1.183085803897985, -2.3827810941365573], m = MyType(1.0, 2)), ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_empty_Tuple#14"{Float64, Tuple{}}, var"#unflatten_to_Any#6"{Float64, MyType}}, ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}}, ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}}(6, 2, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_empty_Tuple#14"{Float64, Tuple{}}, var"#unflatten_to_Any#6"{Float64, MyType}}, ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}}(0, 6, ParameterHandling.var"#unflatten_to_Tuple#13"{Float64, Int64, Int64, ParameterHandling.var"#unflatten_to_empty_Tuple#14"{Float64, Tuple{}}, var"#unflatten_to_Any#6"{Float64, MyType}}(0, 0, ParameterHandling.var"#unflatten_to_empty_Tuple#14"{Float64, Tuple{}}(()), var"#unflatten_to_Any#6"{Float64, MyType}(MyType(1.0, 2))), ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}()), ParameterHandling.var"#unflatten_to_Vector#4"{Float64, Float64}())))

julia> restructure(flatvector)
(a = [0.8409925019315632, 0.2819056254106582], b = [1.598581283953101, 0.3921407396316173, -1.6414335944803775, 1.0503960727007877, -1.183085803897985, -2.3827810941365573], m = MyType(1.0, 2))

Is there any reason for not implementing such default behavior? Am I missing something? Should I make a PR?

[willtebbutt]

Hi @FerreolS . Thanks for opening this.

I think you should probably consider using ParameterHandling.fixed -- this lets you use arbitrary types when you don't want their data to be tracked. Have you considered this?

We don't provide the kind of default method that you suggest precisely because we want users to actively make decisions about how their types ought to be treated, rather than run the risk of silently doing the wrong thing.

[FerreolS]

Thanks @willtebbutt for you prompt answer. The issue with ParameterHandling.fixed is that (if I understand well) the type of the fixed parameter will then be ParameterHandling.Fixed{MyType} and I don't want the package containing the function func in func(restructure(flatvector)...) to depend on ParameterHandling. I would have preferred a kind of noop mechanism as in Adapt where nothing happen unless user has overloaded a function handle its specific types.

Probably ParameterHandling is a bit overkill the simple flatten/unflatten mechanism I need. From #43 I understand that there a no consensus on a lightweight package for that.

[willtebbutt]

Ahh I see. Unfortunately I'm not sure that ParameterHandling.jl is really the tool for this kind of thing. It's intended use is largely not that you would build an arbitrary type, and then be able to flatten / unflatten it. Rather, it is intended to provide a convenient way to assemble simple data structures comprising tuples / named tuples / arrays etc, which are easy to then convert into your desired type.

For example, see the intended style here -- the parameters live in a named tuple, and we have a function (build_gp) which maps from this named tuple to a data structure. So e.g. KernelFunctions.jl doesn't depend on ParameterHandling.jl, but there's a bit of manual work to map from parameters to kernel.