Type Any

[whatsthecraic]

Can you insert another ProtoBuf message inside a field of type Any? If so, how? The google doc refers that this depends on the target language.

[Drvi]

So, type Any contains two fields:

  string type_url = 1;
  bytes value = 2;

value is the protobuf-encoded message and type_url contains the type information of the encoded message. IIUC, there are two ways you can get the message type from the type_url:

a) You can simply parse the fully qualified name of the type from the last part of type_url (e.g. for "path/google.protobuf.Duration" -> you'd go grab the google module, the protobuf submodule and finally the Duration type).

b) The docs mention type-servers that you can hit with GET requests, querying type_url should return another well-known type back, the Type type, which contains metadata you can use to define a new type on the fly.

---

I think we can support a) by providing a custom decode method for Any. Here is a small example:

Let's say there is a struct C defined in a submodule A.B:

module A module B struct C; x::Int end end end

Then, you can do e.g. the following to get the Julia type from a url_string:

function type_from_type_url(x)
    m::Module = @__MODULE__
    i = findlast('/', x)
    @assert !isnothing(i)
    @inbounds while true
        j = findnext('.', x, i+1)
        isnothing(j) && return getfield(m, Symbol(@view x[i+1:end]))
        m = getfield(m, Symbol(view(x, i+1:j-1)))::Module
        i = j
    end
end

get_from_module_manual("abcd/efgh/x/A.B.C")(1)
# Main.A.B.C(1)

# benchmark: 364.380 ns (1 allocation: 16 bytes)

With this, we could provide a special decode method:

function decode(d::AbstractProtoDecoder, x::Type{var"#Any"})
    any_message = _decode_any_type(d)
    T = type_from_type_url(any_message.type_url)
    return decode(ProtoDecoder(IOBuffer(any_message.value)), T)
end

For b), IMO, a lot more design work is needed. We'd need to fetch the Type metadata efficiently, figure out how to construct/represent actual Julia types based on the Type metadada (maybe just NamedTuples?), we'd need a special decode method to work on value bytes using the metadata at runtime.

Frankly, I think it would be better to only support Any for types that are guaranteed to be defined in a Julia session.