Document ad hoc Signal K data model extension mechanism

[tkurki]

Document how Signal K data model can be used even without formal schema extensions to cover data items not included in the model.

https://github.com/SignalK/signalk-parser-nmea0183/pull/53#issuecomment-203045799

[tkurki]

Let's think about what the SK schema is

1. a shared semantic meaning for a key (environment.depth.belowKeel)
2. implied hierarchical grouping (environment and depth in the example above)
3. units (m)
4. human oriented description (Depth below keel)

This allows a SK application to do something about this figure, like construct a UI and select suitable presentation for it, maybe apply functionality like alarms or set colors in the UI or something similar.

Something that we are missing is the fact that depth is more like length than distance - for distance we could provide the option or default of showing it in nautical miles instead of feet.

Now if somebody wanted to add a hypothetical new measurement, say distanceToDock measured by some ingenious device, in meters and under navigation.docking how should the SK data model on that server be extended? Assuming that this is not (yet) a global SK schema addition.

[timmathews]

The distance to dock measuring device could do two things:

1. Provide a definition of navigation.docking.distanceToDock available via HTTP API, e.g. GET /signalk/keyswithmetadata.json ← is there a standard URL for this already?
2. Provide the definition in the hello WebSocket message.

It must do the first, because that supports HTTP-only and WebSocket clients. It should do the second so that WS clients don't need to make the HTTP GET request.

In either case, the format of keyswithmetadata.json needs to be expanded some (maybe).

{
  "~/navigation/docking/distanceToDock": {
    "description": "A novel way to know that it's too late to avoid scraping the crap out of your paint",
    "displayName": "Distance to Dock",
    "shortName": "DtD",
    "units": "m",
    "gaugeType": "digital",
    "$ref": "https://signalk.org/specification/1.0.0/definitions.json#/definitions/numberValue",
    "timeout": 1,
    "alarmMethod": "sound"
    "zones": [...]
  }
}

Essentially, this is all of the keys defined for the meta object, plus description, units and a reference to the Signal K data type.

There is another benefit to this besides vendor specific paths. Eventually there will be a version 2.0 of the schema and we'll have fielded devices which only know about v1.0. When they encounter a newer 2.0 device, they can still download the keyswithmetadata and consume the newer data. This is forward compatibility. IMO, it's a huge selling point.

Some drawbacks that come to mind:

1. We could potentially end up with lots of manufacturers just going off and doing their own thing and ignoring the common published schema. We need to be clear that custom paths are a last resort or for device specific data (configuration, calibration, diagnostics)
2. In cases of apps with complex requirements (e.g. an app that consumes a lot of sailing performance data and makes sailing suggestions or velocity predictions or something) there could be a lot of configuration required on the part of the end user. This could be mitigated somewhat by the app looking for common data points at schema-standard locations and choosing those by default. Then asking the user to fill in the gaps as necessary.
3. More work required for low power sensors. They'd need to be able to store and transmit this configuration data. Manufacturers may need to include more EPROM or flash to store it.

Item 1 is a documentation and communication one. Custom paths are an "advanced" topic and should be documented as such. The documentation should encourage potential implementors to contact the mailing list or developers on Slack to find the best place for their data if it isn't clear.

Item 2 probably won't come up too often in practice because the developer of this fancy app which relies on data at certain paths will either be working with the published schema and use those paths or they'll have advance knowledge of these non-standard paths because they're working for the company which is creating the non-standard paths. I can't imagine a scenario where an app developer would build an app which relied on data being available at navigation.docking.distanceToDock before someone built a device which produced data at that path.

Unfortunately, item 3 is essentially the price of admission for using Signal K in compliance with the specification.

[sbender9]

Unless we have a standard way to get the schema from the server, not sure $ref makes sense.

[tkurki]

keyswithmetadata.json is not part of the SK api now, just a part SK Node internals at this point that came in handy in this context. But it could be.

What we have in the current api is meta structure, retrievable via HTTP api for any keys, also ones that the server has no value for:

• https://github.com/SignalK/signalk-server-node/pull/294
• https://github.com/SignalK/specification/blob/master/gitbook-docs/data_model_metadata.md#metaunits-value

The latter talks just about meta.units, but we could change that easily to cover the whole meta property.

I think it would make sense to provide the same meta structure that we already have.

For connection protocols hello sounds like the way to go, I would not want to force http on everyone because of this. How about periodic transmit of hello for connectionless protocols, like UDP? UDP has the message size limit to think about. Several meta instances could be split to individual messages.