Design questions around the `exploreDirectory` method

[JKRhb]

Recently, I've been dealing a lot with the exploreDirectory method from the Scripting API and its potential implementation. Although we've made some progress both in node-wot and dart_wot regarding the support of the method, I've been noticing at least two challenges for our current architecture that we need to deal with to have proper support in place.

Problem 1: No type in TDD things property

The first problem relates to our current approach of first consuming the Directory TD and then using the readProperty method to access the things property defined in the standardized API. While this has been working so far, the fix for the value() method in #1217 broke the implementation since calling the value() method with a data schema that has an undefined type field is supposed to throw and the things property does not have a type defined. While this is probably also an oversight in the Discovery specification, this does raise the question of whether we should loosen the requirements a bit, both in node-wot and the Scripting API (as also mentioned by @danielpeintner in #1216).

Problem 2: Passing of information from HTTP headers upstream

Another problem that has several implications relates to the fact that pagination information from TDDs is supposed to be transmitted via HTTP headers by default, namely the indication of the offset and limit that should be used for retrieving the next page of TDs. By default (i.e., when using the array format), the directory includes this information in a Link header with a URL that allows accessing the next page (an example would be /things?offset=10&limit=10).

Currently, we would need to control this behavior at the (HTTP) binding level, as there is no standardized way of mapping this header information to a DataSchema at the moment. I think this is a good example of one of the shortcomings mentioned in https://github.com/w3c/wot/pull/1165. A potential alternative could be using the collection format by the way, where the information in question is moved from the HTTP headers to the response body. However, it seems to me as if a directory does not have to implement this format (if I am not mistaken), therefore we don't have the guarantee that this variant will work with every TDD.

The problem with passing the information upstream made me wonder whether we should create a dedicated method in the ProtocolClient interface to handle the exploreDirectory method. However, this would probably lead to a lot of binding implementations only having a method stub that just throws a "Not implemented" error :/

These are just a couple of observations I've made so far which can probably be the basis for further discussions, both on an implementation and use-case/specification level.