emoncms
Increased phpfina precision of stored values by adding a start value (offset)
When storing consumption as a virtual meter reading the precision of the 32bit floats is limiting. This has come up many times on the forum where users have started a feed with their current meter reading rather than at zero and they instantly lose precision due the use of floats.
Could we add a "initial reading" value to the phpfina so that no matter what your existing meter reading is, you still get full range of the 32bit float?
The maths can all be done in the engine classes as the engine would simply offset each stored value by the "initial value", much like it sums the start time and the values file position/4.
There are already 8 unused bytes at the beginning of a meta file, we could use the second 4 bytes to store a 32bit float "initial value", that value is already "0" by default so this feature could be rolled out with no consequence to existing users since value + 0 (initial value) means no change to existing feeds, api's or graphs etc.
Although this affects all the php feeds, the phpfina has a meta file which makes it easy to fix in this instance.
This could also help users align their feeds and meter readings, not only when they create the feed, but following an error, disconnection or just correcting general inaccuracies, they could edit that initial value so that historical values are slightly "off" rather than having a "correction" in the more recent data. This api call, would also then allow existing users (that started thier feeds at 0) to now add an "initial value" to align their meters retrospectively.
Also if we get "rolling" feed length limiting at some point, we can just sum the old "initial value" to the value at the cut-off point and use that as the new "initial value".
This last point could help with ongoing precision by keeping the latest values within a certain range. I have several sites now that have been collecting data for quite a while now and being commercial sites they have a much higher rate of consumption and PV yields, that I now have precision issues despite having started at 0. It would be useful to reduce the feed size by deleting the earlier data, setting a "initial value" and subtracting that value from every remaining datapoint (using post-process) to bring the datapoint values back into a range that isn't loosing precision.
An alternative is to use the mysql engine since the installation is local. It defaults to a 64bits DOUBLE that fits most values cases already. But you could even specify (manually on the database) any formatting like DECIMAL(m,n) to best fit the values range that the feed's data will use.
Thanks @chaveiro, indeed that is a possibility, but it often seems like a huge compromise. The majority of emoncms development is done for phpfina (& sometimes phptimeseries) only, so using anything else feels like you're swimming against the tide. Even the apps module is coded to only work with phpfina. yes, I edit my local copy of apps to work on a wider range, but the sync module and the post-process module do not cater for other feed types either.
There are so many times I have been caught out by this, for example I wanted to do some tests to see how much the timestamps get fudged by network and processing delays, integer truncating, fixed interval feeds etc etc, but I cannot even store a unix timestamp in a phpfina or phptimeseries feed as it gets rounded to the nearest 15seconds or so.
I would like to use mySQL more often and also try influxDB too, but it all seems too difficult to go against the flow.
I have also used virtual feeds to avoid saving the results of summing feeds, not to save disc space, but just to avoid trying to persist large high precision numbers, although it works, the response time for drawing graphs can be affected and again, they are not supported on emoncms.org or in the apps modules etc.
The other quite minor thing that deters me from using mysql feeds is the fact it mingles my feeds with my configuration tables, if the feed data was in a separate table or at the very least, the feeds were prefixed with "z" or "_" etc to group them at the end of the table list I might be tempted to try it again.
I have previously suggested having phpfina of varying sizes/datatypes so that we can have 32bit floats or Unsigned longs (with no change of discspace used) or single byte feeds for "on/off" status feeds (optimizing discspace) or 64bit doubles or even decimals for those important numbers (with a disc space increase) , but they all require substantial changes since the current phpfina engine doesn't provide much scope for expansion so it would probably be a separate engine and still get excluded from mainstream features and development.
I simply thought this suggestion was easily implemented and totally backward's compatible whilst still being phpfina so it doesn't get left in the wings and also gets fully supported at emoncms.org too.
I think it would be worth looking at the option of a 64bit double option for PHPFina. I was looking at the possibility of using a cumulative kwh feed to work backwards to power data on the fly (virtual feed) and realise that more precision is required.
This will be a large feature to implement and the development list is long so we need to put in relevant priority order, e.g STM32 development etc. Im not too keen on the idea of including a starting offset but I can see it could be a quick fix.. but then perhaps a layer of complexity that would be good to avoid..
but I can see it could be a quick fix..
It would also provide a quick fix, the start value offset is useful in it's own right to align meter readings.
The difficulty I see is, how would you create an interface to allow for the setting of the starting value, it would create a lot of complexity in the input process interface.
The difficulty I see is, how would you create an interface to allow for the setting of the starting value, it would create a lot of complexity in the input process interface.
Off the top of my head, how about an edit box in the feeds page, like the "unit" field or "tag" field. There are discussions elsewhere about displaying the start time and interval, start value could also be displayed and editable whilst the other 2 are read-only.
Noting as feature request to be kept open as a reference for future development, when time/resources allow.