The HTTP Query Spec has a `Token` value in it that needs to be either generic or `any`

[hansl]

From the point of view of the spec, this field's type does not matter; whatever the canister returns we send back in the same shape. But from the Candid file documenting the HTTP Query spec, it becomes a simple record {} because there's no way to communicate this properly in Candid.

Either Candid could implement a generic (in which case the token becomes T) or the any type, as the token could change type between two responses (and the spec does not actually care).

[nomeata]

Using record {} or any will cause the Candid decoder to throw away information, so cannot be used for round-tripping. Until Candid supports such parametric types (unlikely), I think your best bet is to use a blob or nat here.

[chenyan-dfinity]

The problem with blob is that in Motoko, we need to expose the serialization/deserialization primitives to the end user. Are we ready to open this up?

The use of record {} is only in one direction. The receiver is expected to use a more concrete type, e.g. record { field: nat } to decode the value.

Another idea is to use an "opaque" type: serialization converts all values to an opaque value, and deserialization decodes the opaque value as a sub-Candid message. Would this work?

[hansl]

Using record {} or any will cause the Candid decoder to throw away information,

any is basically telling Candid message transport to NOT throw information. And in my case, the HTTP specification is telling that whatever this candid value is, don't throw any of it. The point is that whatever you give as a value will be given back to you directly, but the transport layer does not care about its content or shape.

Candid does not have a proper way of communicating envelopes or opaque types. Candid-in-a-blob is a poor way of doing it (and in general very painful).

[nomeata]

any is basically telling Candid message transport to NOT throw information.

I understand that you want that to mean that, but that’s not what would happen.

Candid does not have a proper way of communicating envelopes or opaque types.

It has no way of doing that, and using any doesn't fix that. Essentially, you are specifying the HTTP gateway to behave in a way that doesn’t correspond to a single Candid type. I guess it is in theory implementable, but you’d need to do things like “extract the part of the type table referred to by token that comes from the canister, and include it in the type table in the next message”, which are beyond what our current encoders and decoders support, let alone the metatheory.

So the “whatever-in-a-blob” is probably the better choice at this point. Note that it doesn't have to be Candid in a blob. In most cases – this blob is writen and read by exactly one party, so no need for an interop format here. (In most cases, it would just be a single number indicating an offset anyways, right? Easy to encode as a blob without needing Candid.)

The problem with blob is that in Motoko, we need to expose the serialization/deserialization primitives to the end user. Are we ready to open this up?

I never had any issues with that. But @rossberg might have opinions, too :-)

[hansl]

that’s not what would happen

I just want to point out (it might have been missing from this conversation) that the Candid file in the HTTP Request spec is being used as a reference and should not necessarily copy-pasted around. This is a reference for the transport of values.

I guess it is in theory implementable

Not just in theory. The Rust library implements this as a first class feature with IDLValue.

So the “whatever-in-a-blob” is probably the better choice at this point.

It is not when implementing the canister in Motoko. Serialization isn't good, period. At least with this proposal Motoko can type it however it wants and the HTTP Server won't see anything.

let alone the metatheory

The metatheory would dictate that we use generics, which is what this issue is about. I don't know how to apply more pressure and not implement something that will be a breaking change when generics come. This was the best of both worlds and the most pragmatic solution to this problem.

[nomeata]

that the Candid file in the HTTP Request spec is being used as a reference and should not necessarily copy-pasted around.

Then maybe don't write it as a Candid type (because any is just wrong), but make it a “Candid type template”, saying that “the service should have interface … token : t… for some freely chosen Candid type t”. Then it's clear that any service implementer should pluck in their concrete type before feeding this to any kind of Candid tooling (stub generators, type checkers etc.)

I guess it is in theory implementable

Not just in theory. The Rust library implements this as a first class feature with IDLValue.

I am not sure if this is enough, i.e. if an IDLValue value has enough information about its type. E.g. an empty vector, or a actor reference. (Ok, for the empty list sending it as vec empty would work, i.e. there is a principal type, but that's not true for reference types.) Anyways, we'll see what happens.