Support custom structs

[rana]

The performance looks great and am open to trying JDF.jl.

I happen to be using custom structs in DataFrames and serializing with JLD2.jl.

Have you considered supporting custom structs?

Fwiw, not sure if JSON3.jl usage of StructTypes.jl would help you support custom structs.

[xiaodaigh]

It supports custom structs already as long as the type is isbits or is made up compatible types

[xiaodaigh]

Maybe show me the custom struc definition. You can define your own savers with JDF. Just not documented yet.

[rana]

The DataFrame looks like

  trials = DataFrame(
    trial=Int32[], epch=Int64[],
    nn=NeuralNetwork[], nn2=NeuralNetwork[]
  )

where NeuralNetwork is

abstract type NeuralNetwork end
abstract type RecurrentNetwork <: NeuralNetwork end
struct Chn{R,D} <: RecurrentNetwork
  fwd::R
  dns::D
end
function Chn(in::Integer, hdn::Integer, out::Integer, σ::Function=identity)
  Chn(
    RNN(in, hdn, σ),
    Dense(hdn, out, σ),
  )
end

Maybe savers could be the answer to saving custom structs; though, I think that is more work than I would be willing to do when JLD2 works fine, even if not as efficient. Am mostly looking for a DataFrame storage solution that I don't have to think about. So that I can focus on other problems :)

I like JDF, thank you.

[xiaodaigh]

Am mostly looking for a DataFrame storage solution that I don't have to think about.

Totally understand. The reason I made JDF is for ease of use and stability. It can not handle arbitrary types, because there are a potentially limitless amount of issues. As a compromise, I am open to saving unknown column types using JLD2 automatically to smooth the process a little. But I wanted to this to be cross-language too, so doing that comes with baggage too. But cross-language is just a dream atm. So would prioritise practical features.

[xiaodaigh]

Actually, apart from the $\sigma$ function, all the rest should be easily serializable with JDF because it's just matrices and numbers.

But JDF can't guarantee to serialize functions. I will have to think about that.