Request: a `metadata` function

[briochemc]

I'm not sure if this is doable, but it would be nice to have a metadata function that would take a struct and return its metadata bits, if any... Starting from the example in the ReadMe,

@metadata describe ""

@describe mutable struct Described
   a::Int     | "an Int with a description"  
   b::Float64 | "a Float with a description"
end

d = Described(1, 1.0)

Is would be nice to be able to extract metadata, i.e., something like

julia> metadata(d)
Described
  length 2
  Symbols: :a, :b
  Types: Int64, Float64
  Metadata: 
    described: "an Int with a description", "a Float with a description"

Maybe it is possible to define a function like

metadata(::Described) = # some info to display

in the same way that, say, the decribed function is defined?

I think I could use that sort of thing to save d into, say, a table, so that it can be easily reconstructed in another session (or simply displayed nicely). But again, not sure this makes a lot of sense, just throwing the idea out there 😄

[rafaqz]

I was actually thinking of adding a Tables.jl interface so we can use model objects directly as tables, and they will print as tables etc. Edit: it would even work for nested components using Flatten.jl

We could maybe find all the metadata methods defined for Described using methodswith(Described, Val{first(fieldnames(Described))}), which should achieve what you want.

[briochemc]

Oh that sounds great! I could definitely use a table to parameter function and vice versa!

I'm not sure about the idea you suggested, I could not get it to work (did not know about methodswith either). However methodswith(Described) returned an empty vector.

[rafaqz]

Hmm I think it needs to be methodswith(::Type{<:Described}). And you can't provide a second argument... It might be a bit hacky really. How about just passing in the list of functions you want to show as in FieldDocTables?

[briochemc]

Do I understand correctly that FieldDocTables is a tool to be used by other packages to display their FieldMetadata types in their documentation?

Back to the table interface. Assuming I am using the Described object above, I would love to be able to

1. convert it into a table (any format is good for me, so I guess having a Table.jl interface would suffice. Then I can use, e.g., DataFrames, PrettyTables, or something else, to make it into a concrete table object, via some table function, maybe something like

   julia> d = Described(1, 1.0)
   Described(1, 1.0)
   
   julia> t = table(d, DataFrame)
   2×2 DataFrame
   │ Row │ symbol │ value │
   │     │ Symbol │ Real  │
   ├─────┼────────┼───────┤
   │ 1   │ a      │ 1     │
   │ 2   │ b      │ 1.0   │
   

2. Save that table to a csv file (that's a DataFrames issue in this case)

3. Load the csv file into a table (that's again a DataFrames issue)

4. convert the table back into the Described type. Maybe something like

   julia> t
   2×2 DataFrame
   │ Row │ symbol │ value │
   │     │ Symbol │ Real  │
   ├─────┼────────┼───────┤
   │ 1   │ a      │ 1     │
   │ 2   │ b      │ 1.0   │
   
   julia> d2 = reconstruct(t, Described) 
   Described(1, 1.0)
   

Again I apologize if these ideas are not clear! 😄

[rafaqz]

Yeah that's what FieldDocTables does, as well as the field doc and the field name.

The tables idea is pretty much what I want to do as well. Converting to tables is pretty easy if you pass the methods that you want as columns (again see FieldDocTables). But the problem is converting the dataframe back to metadata... as metadata isn't stored on the struct or a global variable, but in the method table. And methods can only be added in global scope - so we can't do it with a function, it has to be in a macro, so we can't use run-time data.

This is the main flaw in this whole design. FieldMetadata.jl was designed more for having no runtime overhead - for use in Flatten.jl and similar - than for maximising flexibility. It may still prove to be a generally dumb idea - I don't think any other language has even allowed this kind of thing to work before, lol.

[briochemc]

I get confused with these scopes... So when I do

@metadata describe ""

@describe mutable struct Described
   a::Int     | "an Int with a description"  
   b::Float64 | "a Float with a description"
end

it generates a function called describe with methods for objects of type Described. Why couldn't it also generate a function called, e.g., metadatafunctions, that would return all the metadata functions. Something like

metadatafunctions(::Described) = [describe]

[rafaqz]

Edit: reading your code again: are you not talking about rebuilding the metadata, just the struct values?

That is totally doable. But it wouldn't live here, instead it would be over in Flatten.jl. It would also work for tuples and complex nested stucts. What doesn't exist now is a way of checking that the names actually match. I'm thinking of adding a NamedTuple flatten that flattens names and values and checks them on reconstruct - so we would jsut convert the table to a named tuple and use that.

The function metadatafunction is actually a bit tricky - as none of the macros actually know about each other, they just get passed through - each one seeing the code as if it's the outside macro. so they would all write over metadatafunctions until it only contained one method.

Edit: I guess they could call their own metadatafunctions method and get the union of the result and the current method... Ok this can work.

Thanks for pushing that though :)

Although its kind of insane use of code generation, but we've come this far...

[briochemc]

Oh sorry no I meant the metadata —I messed up the code — I meant something more like

julia> d = Described(1, 1.0)
Described(1, 1.0)

julia> t = table(d, DataFrame)
2×3 DataFrame
│ Row │ symbol │ value │ describe                     │
│     │ Symbol │ Real  │ String                       │
├─────┼────────┼───────┤──────────────────────────────┤
│ 1   │ a      │ 1     │ "an Int with a description"  │
│ 2   │ b      │ 1.0   │ "a Float with a description" │

And I don't want to push for anything crazy 😄 I'm just unable to anticipate what's doable, useful, easy to implement, etc., and wha'ts not... So I'm just throwing ideas. I thought I could use a metadatafunction to construct the table, but I am not sure at all at this point...

[briochemc]

BTW I should let you know that I have been doing this the otherway until now. That is, I first create a table and I create the struct from the table. Maybe that was the best way to go? I just don't like some aspects of the interface (of my package).

Anyway, after examining your work in FieldDocTables I figured I could use something similar, like

function table(p, nf::NamedTuple) # p is the FieldMetadata-constructed struct, and nf stands for "name" and "function"
    t = DataFrame(symbol = collect(fieldnameflatten(p)), value = flatten(p))
    for (n,f) in zip(keys(nf), nf)
        setproperty!(t, n, collect(f(p)))
    end
    return t
end

where as you can tell I am also using Flatten and DataFrames... I think I can get away with this for now :) But I will leave this open and let you decide when you want to close it! 🙂

[rafaqz]

metadatafunctions is doable, but loading metadata isn't :)

In terms of loading from dataframe - its much better get the field values as a tuple and update an existing struct using Flatten.jl. That lets you use immutable structs in your models, which should nearly always be a good performance boost.

[briochemc]

I think I wanted a mutable struct so that I could make it a subtype of an AbstractVector for those struct to be usable as vectors in things like ForwardDiff, but maybe I don't need it — I am very confused about what I should be doing or not for that anyway...

[rafaqz]

You should be able to use Flatten.jl for that too... It's probably faster, even though it seems more complicated.

Edit: or Setfield.jl

[briochemc]

You should be able to use Flatten.jl for that too... It's probably faster, even though it seems more complicated.

I am not sure I understand. Right now, I define Base.setindex through setfield for my FieldMetadata-generated mutable struct. (I think this was needed for ForwardDiff, which may use setindex but I'm not sure anymore.). So that for a function f(x) (where x is the FieldMetadata-generated mutable struct) could treat x like a vector and ForwardDiff would be able to compute, e.g., a Jacobian... But you are saying I can forgo using a mutable struct and defining my own setindex by using Flatten instead? Could you expand a bit on that?

Setfield.jl looks interesting too!