specification
specification copied to clipboard
History / time series API
We need an API that provides ways to
- discover what time series are available => present a list of for example dates / time ranges for which we have data
- a way to query & summarize the data, for example
Soundings for a date
- September 15
- 5 minute resolution
- position (average), depth (min/max/avg)
Depth data for an area
- all data within a geofence
- position+depth, all data
True wind history
- last 2 hrs
- true wind direction and speed, 1 minute average
Events history
- annotated events created by user inputs
There are undoubtedly countless others.
Here the schema is less important than the query language. For example the current delta format in an array or something similar would work pretty nicely - a collection of data items with timestamps.
From slack discussion:
Basically we can store a list of {"timestamp":"123456", "path":"y.z.x", "value":"nnn"}
in a persistent db. That would provide a place for history outside the signalk model, which is really the 'current' data.
Then I would propose /signalk/v1/history/[context]/[path]?start=xxxxx&end=yyyyyy
So as similar as possible to the REST api, but with its own url. Then we can advertise it if the server supports it. And an equivalent signalk message for other transports:
{"context":"vessels.self","history":{"start":nnnnnnn, "end":yyyyyyyy, "path":"x.y.z"}}
The reply should be either a series of update messages, or a list of the stored objects
{"timestamp":"123456", "path":"y.z.x", "value":"nnn"}
Seems the updates would be more verbose but handy for replays etc.
Your {"timestamp":"123456", "path":"y.z.x", "value":"nnn"}
look awfully lot like a delta message. I think there is value in using one format for both delta stream and history query result. Also a co moremplex use case like position and speed for the racing fleet, 5 minute average would fit pretty well if not optimally to the result being a list of deltas.
I would not put context and path into the url path. How do you query for multiple values? It implies a hierarchy, which is not there. They are also somewhat orthogonal: context is part of the where
clause and path
a part of the select
clause in sql terms.
Food for thought: http://eagleio.readthedocs.org/en/latest/reference/historic/jts.html
Food for thought: http://eagleio.readthedocs.org/en/latest/reference/historic/jts.html
https://github.com/james-hu/cjtsd-js/wiki/Compact-JSON-Time-Series-Data
https://github.com/michaelwittig/fliptable/blob/master/README.md
@vyacht did you have time to look at the resources linked above? Your thoughts on the history api?
Sure, they were the first 3 google hits and I looked at them last week. jts looks natural. Compact format seems a bit awkward - I think you want something more human readable. {"timestamps":[],"values":[]}
is a bit weird as well, separating things that clearly belong together. Other references are GPX, KML for how to present tracks and data collected along the way.
I have been testing more approaches and have a prototype graph running. I seem to always end up with something close to my PR. Moved time references directly under vessel for fun and using arrays "vessel":[{"navigation":{}},{"navigation":{}}]
. Basically generating a track of all data for each boat as an array.
Some thoughts
- Switch from historic data to most recent updates - this comes in handy when you want to draw a depth graph, pull the history and then continue with the new data
- Pagination
- Data often makes more sense in a context of multiple values and not only time (e.g. depth at position).
- "Aggregates" (max, min, mean), various periods (historic, last, next, last1h, last24h) and also boxing data (max and min lat covered by data) for e.g. tidal data, speed, wind speed
- and 2. are covered by the above. Better to use wording like "startAfter" instead of "start" for pagination.
- could be solved by
"vessel":[{"navigation":{}},{"navigation":{}}]
- remains difficult
Approaches for query APIs are sufficiently well understood these days. REST example:
?startAfter=xyz&fields=navigation.depth,navigation.position
https://docs.influxdata.com/influxdb/v1.0/guides/querying_data/
Pushing SK numeric data to InfluxDB is pretty straightforward and visualisation via http://grafana.org/ is pretty easy.
The artemis-server stores all signalk data directly in a timeseries db (currently influx) natively, and uses realtime query of the data for all output. See https://github.com/SignalK/artemis-server
The current behaviour uses latest timestamp for any requested key which is consistent with other signalk servers behaviour. However there is no reason not to request data at any point in the past.
As noted by @vyacht we can send the request easily by REST or signalk subscribe
or get
message.
This would just require a new parameter (timeAt
?)to pass the requested time in the request and still be fully compatible with existing signalk output. Since we already include timestamp
in messages the resulting message would just have a timestamp
in the past.
Requesting values across a time span can be done in a similar way with timeFrom
and timeTo
parameters. I think the delta format will allow these to be returned, since each value has a timestamp
.
From work with grafana it seems we would also need an interval
(or resolution
#363 ) key, so the data can be summarised to 1sec, 1 minute, etc. We dont want a 10Hz value over 30 days!.
That would also require an 'aggregate',key eg avg
, mean
, max
, min
etc,
Using existing formats may not be the most compact, but does have the advantage of minimal change to existing apps. By incorporating a requestTime
widget most could easily move back and forth to any historic point, and by adjusting the subscribe behaviour to begin in the past, they can also do replays.
That leaves the concept of filters
or where
clause, but I think thats a complex issue, and can be added later.