Add support for bound function arguments

[rossberg]

This specs out the Candid side of the "closure" extension I suggested here.

WDYT, does this work?

Edit: see also #292, for a complementary extension with type dynamic.

[matthewhammer]

LGTM!

My only nit is about the other companion PR, and how to compose this PR with that one in a larger motivating example.

Let's say that the wallet call wants to return some data, and we don't know what it is, or even its arity?

How would we adapt this?:

type wallet = service {
  topup : (amount : nat) -> ();
  forward : (call : () -> ()) -> ();
}

I see three options for the non-unit return type of call and forward:

1. Blob
2. dynamic (see #292)
3. a variant of dynamic that I will call dynamics (e.g., see comment https://github.com/dfinity/candid/issues/245#issuecomment-959449726)

1. Use Blob

forward : (call : () -> Blob) -> Blob;

And then assume that this Blob holds a candid response sequence, not a single candid value.

Pros: Blob exists today in Candid and Motoko. Cons: Blob conversions are explicit, and perhaps not ergonomic yet. They could also have type errors.

2. Use dynamic (see #292)

forward : (call : () -> dynamic) -> dynamic;

Pros: Could be a safer option to option 1, since conversions would have dynamic checking and auto trapping (I presume). Cons: Unfortunately, this API is not quite as "general" as the first, since dynamic cannot encode a sequence of values, but only one value. See option 3.

3. Use dynamics (see comments to #292)

forward : (call : () -> dynamics) -> dynamics;

Pros: Actually solves the problem (I think). Cons: Furtherest of the options from what we have today.

[rossberg]

For the time being I think it's fine if this pattern only worked for functions of fixed arity. If we wanted variadic abstraction, maybe we could make a tuple record coercible to a parameter list and vice versa, analogous to what we do in Motoko.

Perhaps n-ary functions as a primitive in Candid were a mistake (as @nomeata always argued, though for different reasons). If we introduced such a coercion, could we retroactively pepper over it?

[rossberg]

@nomeata, fair points about constructing the new call.

I would assume that the serialiser would not merge but merely extend the type table it gets from the closure. Since we allow duplication, it could do so blindly.

For the argument tuple, if it extended the type table from the closure as just said, then I agree that it suffices to just copy over the serialised blob for each value. (We could even add an leb128 with the length for each value's encoding to the TM function I now introduced; however, I'm not sure we want that – for security reasons, the deserialiser should validate the data anyway to the extend it can, and could separate the individual values right there.)

So yes, some work is required, but it doesn't seem too bad?

(With hindsight, I actually think it was an oversight that our function types did not allow closures from the beginning. It seems like a glaring omission for a higher-order data format.)

[nomeata]

But that would require a backwards-incompatible change to the encoding of calls, wouldn't it?

Oh, right, of course. But that means we have an impossibly result: we can't merge type tables (because of future types), but we also can't keep them separate (because it wouldn't be backwards compatible). This seems to prevent currying…

[nomeata]

(For some reason I can't mark conversations as resolved here)