mp-se
Add "Gravity Velocity" as a measure
Is your feature request related to a problem? Please describe. Gravity measurements at individual timepoints may provide a basis for doing automations, they still fluctuate and wouldnt be too reliable to understand the current state the fermentation is in. I understand that calculating the fermentation state by understanding how many gravity points per day the ferment is dropping is a common practice but its a manual effort if you dont have an app or script that does the job for you after you get the gravity measurement from a hydrometer. Kegland recently posted a nice video on this and i rly like the idea to have the hydrometer do the work for you.
Describe the solution you'd like I therefore propose adding such a measure to Gravitymon as well. This would greatly benefit automating fermentations even further.
Proposal how to measure this (also based on what Kegland discloses in their video, and some considerations I made):
- Measure in points/day
- Take hourly reading of current gravity
- Fit a linear curve with least squares to reduce outliers
- Sliding window for measurement points: 6h, 12h, 24h as options. Calculation can start after 2 measurments, in this case 1h, but gets more precise reaching end full window
- Calculate gravity velocity hourly based on the current curve for timeframe of 24h to get points/day
To avoid weird values in the beginning of fermentation:
- Measure starts at 0
- If measure is negative, set to 0
Describe alternatives you've considered Using nodered to deal with it. Its not a great user experience and requires tinkering and e.g. using nodered as a proxy to send the current reading including this measure to any integration/service.
Additional context Reference video from Kegland: https://youtu.be/oVNjjFeZWxk?si=D7PUPJszUSMETb7k
Thx a lot for considering the suggestion. :-)
HI, thanks for the feature request and it would be an interesting feature to have. I see a few challenges on implementing this on the device itself,
- The device does not do any timesync so I dont know when the data was captured. Adding time sync is possible but would reduce the battery power quite a lot so this is a feature that I have dropped in the past.
- Will require writing data to the device that will reduce the lifespan of the flash (has limited writes)
- Finding some good libraries for doing the math and calculations, need to investigate how much time these calculation would require.
- Having something that can read and show the value submitted. Most services limit use to the standard iSpindle format...
My first thought was that this feature would be much easier to implement in a server side (receiver) software and there show the trends and data (you will still need something that can display it). I guess kegland has their own app or service that does this.
However I will investigate this further and try it out server side first. I have another project I use for collecting data from my iot projects which I can test out the principle first to see if its feasible to do on the device.
Thx a lot for the consideration! I agree that there are some challenges. Some consderations from my side on your comments.
The device does not do any timesync so I dont know when the data was captured. Adding time sync is possible but would reduce the battery power quite a lot so this is a feature that I have dropped in the past.
I think we dont need timesync in this case. We need to have measurement points every 3600 seconds. The timer can be used similar than already in use for sending every X seconds to the integrations. From there we can count measurement points to understand how many hours passed for e.g. sliding window. Points per day can also be calculated "sliding" meaning we dont need specific points in time, just relative.
Will require writing data to the device that will reduce the lifespan of the flash (has limited writes)
Agreed. Could we keep this just in RAM? I am not completely sure about deep sleep but i think it preserves RAM right? I mean, if the hydrometer shuts off completely then measurement process starts from fresh. After a period of time it gets the same accuracy as before it turned off. I dont think that this is a big issue. between fermentations its fine to turn if off and start fresh. If this happens while fermenting its an issue anyway if the device resets due to any issue or turns-off/on again.
Finding some good libraries for doing the math and calculations, need to investigate how much time these calculation would require.
I can look more into the math in general. I was more conceptualizing this at the moment and trying to envision how to replicate it, and put my thoughts in. I can work it out a little more, but I am not any good in writing C. I could do it in python though ^^
Having something that can read and show the value submitted. Most services limit use to the standard iSpindle format...
Indeed. I think this one rly depends on the respective integrations. For generic HTTP/S and MQTT it works i think with no issues. Maybe a check box to incude/exlude it would be an idea. For other "fixed" integrations i think it depends if a completely fixed JSON object is expected, respectively if integration breaks on receiving end if additional key/value pairs are sent. Brewfather custom stream should e.g. not be an issue at all. This one might require more assessment in general and testing though. I could only test a few like http, mqtt and brewfather.
My first thought was that this feature would be much easier to implement in a server side (receiver) software and there show the trends and data (you will still need something that can display it). I guess kegland has their own app or service that does this.
Was exactly my first thought as well and just do it in nodered. However, sometimes integrations on receiving end also change due to changes on sending end. So also possible, after sth like that is implemented to raise enhancement requests there as well. Regarding Kegland, i would suspect they do this on the device as well but of course there is a chance its server calculated. Some settings for this measurement are on device settings though as far as i can see. I dont have one to test, but they offer bluetooth and wifi. If i had one, i could check the bluetooth data to see if its added there or not.
Thx again for considering :)
I think we dont need timesync in this case. We need to have measurement points every 3600 seconds. The timer can be used similar than already in use for sending every X seconds to the integrations. From there we can count measurement points to understand how many hours passed for e.g. sliding window. Points per day can also be calculated "sliding" meaning we dont need specific points in time, just relative.
Well there is no guarantee that measurements will be in a fixed interval, connection issues etc can throw that off. I dont know how much that would affect the calculations, but it needs to be considered. The only way to know the interval is to have timestamp which means an NTP sync is required.
Agreed. Could we keep this just in RAM? I am not completely sure about deep sleep but i think it preserves RAM right? I mean, if the hydrometer shuts off completely then measurement process starts from fresh. After a period of time it gets the same accuracy as before it turned off. I dont think that this is a big issue. between fermentations its fine to turn if off and start fresh. If this happens while fermenting its an issue anyway if the device resets due to any issue or turns-off/on again.
There would not be enough memory on an esp8266, its already at the limitations as of now. Ram is no preserved while doing a deep sleep. its basically a full restart of the device. so writing to Flash is the only option.
I can look more into the math in general. I was more conceptualizing this at the moment and trying to envision how to replicate it, and put my thoughts in. I can work it out a little more, but I am not any good in writing C. I could do it in python though ^^
If you want to help with the calc that if fine, python is perfect. My data collector is written in python. So that would be a good place to test it. Javascript could also be an option depending on if this is UI presentation feature or needs to be persisted in the backend.
Indeed. I think this one rly depends on the respective integrations. For generic HTTP/S and MQTT it works i think with no issues. Maybe a check box to incude/exlude it would be an idea. For other "fixed" integrations i think it depends if a completely fixed JSON object is expected, respectively if integration breaks on receiving end if additional key/value pairs are sent. Brewfather custom stream should e.g. not be an issue at all. This one might require more assessment in general and testing though. I could only test a few like http, mqtt and brewfather.
Home Assistant and others can display the data but i dont see that brewfather or similiar can handle.
Was exactly my first thought as well and just do it in nodered. However, sometimes integrations on receiving end also change due to changes on sending end. So also possible, after sth like that is implemented to raise enhancement requests there as well. Regarding Kegland, i would suspect they do this on the device as well but of course there is a chance its server calculated. Some settings for this measurement are on device settings though as far as i can see. I dont have one to test, but they offer bluetooth and wifi. If i had one, i could check the bluetooth data to see if its added there or not.
I think kegland has its own cloud service so its likely that they do the calculation there. But I see one option and that is to put this logic in the gravitymon-gateway that is always running or in the brewlogger software that I'm working on. Links below
- https://github.com/mp-se/brewlogger
- https://github.com/mp-se/gravitymon-gateway
Took me a while to respond, didnt have too much time.
I think kegland has its own cloud service so its likely that they do the calculation there.
I looked a bit into that. I still think its calculated on then device somehow. See below. From firmware 2.0 onwards, gravity velocity is part of the BLE packet that is sent out. Also in BLE only mode.
https://gitlab.com/rapt.io/public/-/wikis/Pill%20Hydrometer%20Bluetooth%20Transmissions
There would not be enough memory on an esp8266, its already at the limitations as of now. Ram is no preserved while doing a deep sleep. its basically a full restart of the device. so writing to Flash is the only option.
Very valid. Maybe this is down to the way how the manage energy consumption. Could be they dont use the traditional "deep sleep" method. Could be that they actually limit energy consumption in a different albeit not so effective way. IIRC battery run time on wifi is less on the pill than on sth like iSpindle and Gravitmon. Additionally, probably uses esp32 or sth comparable. But i could be incorrect.
In any case, when I have some more time, i will look into a proposal for calculation and post it. I dont have any "urgency", was more of a nice to have anyway :-)
I did some quick analysis on some of my previous brews and I dont think they do something advanced such as linear regression. I believe they just calculate how many points the gravity is changing per day (or time period).
For example the gravity drops from 1.0341 to 1.0318 a change of -3.30, this is probably the value that is reported. The value if one do a linear regression on the same data would be -4.258468889416096e-11
If this is the case then the amount of data that is stored would be of great significance so that the calculation is not totally off. Even faulty data values would have big impact on that calculation. So how to avoid having faulty measurements impacting the calculation would be a challenge.
The pros is that it does not really require any advanced linear regression library (even i if I have found one that works).
Here is an example from my last brew.
Did some more research on the topic and this is what copilot reported as the measurement;
_Gravity velocity in the context of the RAPT hydrometer refers to the rate of change of specific gravity over a specified interval. This measurement helps brewers monitor the progress of fermentation by indicating how quickly the gravity is changing, which can signal different stages of the fermentation process.
To calculate gravity velocity, the RAPT hydrometer measures the specific gravity at regular intervals and then determines the rate of change. This is typically expressed in points per day, where one point equals 0.001 specific gravity (SG). For example, if the gravity changes from 1.050 to 1.040 over 24 hours, the gravity velocity would be 10 points per day._
I have added the logging part of the gravity readings (10 of them) but i will need to do some more testing on the calculation part since i don't store the full 24 hours of data. Some kind of curve fit is probably a good idea since that would get rid of the readings that is way off. Then one could extrapolate how the data was 6, 12 or 24 hours ago and then use that to calculate how much has changed.
I have done some more testing on this feature based on data I have stored from previous brews.
The approach I took was to take a few datapoints and do a polynomial regression and then try to extrapolate what the data was 1 day ago (or x number of data points ago since I don't have timestamps but know the interval).
Currently I need a last 30 data points to get some reasonable results which is quite a lot to store on the device, it would be preferable to get down the amount of data needed for the calculations to less than 10 data points.
It might be possible to get by with less if I can filter out readings that is invalid (see the image above for spikes). So this would be my next steps.
Any suggestions is appreciated.
Below you can see the result and the gravity velocity is between 6 and 0 points per day.
The larger the window the better prediction.
With the current approach and the need for a large window of data I dont see that this would be feasible to implement on the device side, the minimum window is around 15 points with the current algorithm. I have a version implemented in my Brewlogger software. If someone can find a good approach for this I would love to add this to the software. Putting this on hold for now.
@Levi--G do you think this could be feasible with the modern gyros?
The modern gyros are more accurate, so they might produce better values with smaller window sizes and linear regressions might be more accurate/usable, but i personally don't really have a use for it since my software calculates all that so the gravmon doesn't need to.
Technically its possible though on esp32 by using rtc mem, or even without if using a fifo chip since it has x minutes of data it can calculate a linreg on. I see 4 methods if you want to do it:
- Use x previous samples to calculate the linreg, might lag behind a bit (would need rtc mem so esp32 but can work on mpu as well)
- Use Fifo samples to calculate linreg, immediate result but needs a newer chip and might have some jitter due to shorter window time
- Wake esp more, to collect more data for linreg, but only send every x wakes, needs rtc mem and will use a lot more power (i don't recommend it but it will most likely be the most accurate)
- Do it externally, some cloud providers do it already, for those who don't we could make a small "server" or gateway in between that calculates all you want and then passes the data on with a similar config as the gravmon now. I actually made something like this to sit between my gravmon and my homeassistant setup so i could auto register the device and have movavg filtered results. Its not modular atm though so i havent posted the code on github as it is atm tailored to my taste.
Before using more rtc mem for this i would suggest first trying to keep the gravmon settings in rtc memory though, as that will take up quite a bit, have big gains in battery and speed and there might not be much left on certain esp32 chips after that.
@Levi--G do you think this could be feasible with the modern gyros?
The modern gyros are more accurate, so they might produce better values with smaller window sizes and linear regressions might be more accurate/usable, but i personally don't really have a use for it since my software calculates all that so the gravmon doesn't need to.
I have implemented it in my brewing software as a test, but I need around 15+ data points for a resonable calculation with the data I have stored so far, especially if there is a very active fermentation....
Technically its possible though on esp32 by using rtc mem, or even without if using a fifo chip since it has x minutes of data it can calculate a linreg on. I see 4 methods if you want to do it:
- Use x previous samples to calculate the linreg, might lag behind a bit (would need rtc mem so esp32 but can work on mpu as well)
- Use Fifo samples to calculate linreg, immediate result but needs a newer chip and might have some jitter due to shorter window time
- Wake esp more, to collect more data for linreg, but only send every x wakes, needs rtc mem and will use a lot more power (i don't recommend it but it will most likely be the most accurate)
- Do it externally, some cloud providers do it already, for those who don't we could make a small "server" or gateway in between that calculates all you want and then passes the data on with a similar config as the gravmon now. I actually made something like this to sit between my gravmon and my homeassistant setup so i could auto register the device and have movavg filtered results. Its not modular atm though so i havent posted the code on github as it is atm tailored to my taste.
Before using more rtc mem for this i would suggest first trying to keep the gravmon settings in rtc memory though, as that will take up quite a bit, have big gains in battery and speed and there might not be much left on certain esp32 chips after that.
Using RTC memory would be a good option to test out at least. But I will collect some data first to see what is needed. If the data quality is good, then you should only need a few samples for the calculation.
Thanks for your comments, this is on hold for now at least until I have collect enough data to do some testing on data collected. I hope to do the first real tests with the new gyro this weekend.
Will be added to 2.2.0 beta 3 but only available on ESP32 using RTC MEM. Storing data in the flash memory on an esp8266 will reduce the lifespan of the device to much.