FlowFuse
Is there a way to have the data in the 2.0 chart node survive restarts?
Description
The V1 chart has a tested and working subflow that saves the data and reloads the data after a restart. This is pretty vital functionality and needed for V2 charts. Can someone have look at this when there is time please?
I tested the old working subflow with the V 2 chart and as expected it does not work. It would be great if this subflow could be updated to work with 2.0. or of course any other way to have the data survive reboots if possible? This is the subflow that worked with V1 hooked up to a V2 chart for tests:
[
{
"id": "3b6a1337b69e1574",
"type": "subflow",
"name": "vvb d2",
"info": "",
"category": "",
"in": [
{
"x": 50,
"y": 30,
"wires": [
{
"id": "0803c4cc5eb6de01"
}
]
}
],
"out": [
{
"x": 820,
"y": 160,
"wires": [
{
"id": "1d445f73ea677ec2",
"port": 0
},
{
"id": "de6cdd9bd32b3eb1",
"port": 0
}
]
}
],
"env": [],
"meta": {},
"color": "#DDAA99"
},
{
"id": "1d445f73ea677ec2",
"type": "file",
"z": "3b6a1337b69e1574",
"name": "",
"filename": "",
"appendNewline": true,
"createDir": true,
"overwriteFile": "true",
"x": 660,
"y": 140,
"wires": [
[]
]
},
{
"id": "006debb5e922bb46",
"type": "file in",
"z": "3b6a1337b69e1574",
"name": "",
"filename": "",
"format": "utf8",
"sendError": true,
"x": 500,
"y": 180,
"wires": [
[
"de6cdd9bd32b3eb1"
]
]
},
{
"id": "e37d519cbbd162d3",
"type": "json",
"z": "3b6a1337b69e1574",
"name": "",
"x": 500,
"y": 140,
"wires": [
[
"1d445f73ea677ec2"
]
]
},
{
"id": "de6cdd9bd32b3eb1",
"type": "json",
"z": "3b6a1337b69e1574",
"name": "",
"x": 660,
"y": 180,
"wires": [
[]
]
},
{
"id": "0803c4cc5eb6de01",
"type": "function",
"z": "3b6a1337b69e1574",
"name": "LoadSave",
"func": "var strSafe=msg.topic;\nif(strSafe)\n{\n strSafe=strSafe.replace(/[^a-z0-9]/gi, '_').toLowerCase();\n var loaded=context.get(strSafe)||0;\n msg.filename =\"/home/hb19/.node-red/flowsave/vvbd2/\"+strSafe+\".dat\";\n //node.warn(loaded);\n if(0===loaded)\n {\n //load chart!! \n msg.payload=\"load\";\n context.set(strSafe,1);\n \n //node.warn(\"Loading \" +strSafe);\n }\n else\n {\n //node.warn(\"Writing \" +strSafe);\n }\n return msg; \n}\nreturn null;\n\n",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 200,
"y": 160,
"wires": [
[
"abd9aa795d47fbc6"
]
]
},
{
"id": "abd9aa795d47fbc6",
"type": "switch",
"z": "3b6a1337b69e1574",
"name": "Load data",
"property": "payload",
"propertyType": "msg",
"rules": [
{
"t": "neq",
"v": "load",
"vt": "str"
},
{
"t": "eq",
"v": "load",
"vt": "str"
}
],
"checkall": "false",
"repair": false,
"outputs": 2,
"x": 350,
"y": 160,
"wires": [
[
"e37d519cbbd162d3"
],
[
"006debb5e922bb46"
]
]
},
{
"id": "3dc8c1a1d345ef1d",
"type": "ui-chart",
"z": "98b31619.daed3",
"g": "120d335300e6531b",
"group": "f83d15d3a238accd",
"name": "vvb chart week",
"label": "",
"order": 4,
"chartType": "line",
"category": "topic",
"categoryType": "msg",
"xAxisLabel": "",
"xAxisProperty": "",
"xAxisPropertyType": "property",
"xAxisType": "time",
"xAxisFormat": "",
"xAxisFormatType": "auto",
"yAxisLabel": "",
"yAxisProperty": "",
"ymin": "15",
"ymax": "65",
"action": "append",
"stackSeries": false,
"pointShape": "line",
"pointRadius": "2",
"showLegend": false,
"removeOlder": 1,
"removeOlderUnit": "604800",
"removeOlderPoints": "",
"colors": [
"#ff9300",
"#00fdff",
"#ff7f0e",
"#2ca02c",
"#a347e1",
"#d62728",
"#ff9896",
"#9467bd",
"#c5b0d5"
],
"textColor": [
"#ffffff"
],
"textColorDefault": false,
"gridColor": [
"#7a7a7a"
],
"gridColorDefault": false,
"width": "30",
"height": "10",
"className": "",
"x": 4420,
"y": 540,
"wires": [
[
"2e85ceab3abb96f6"
]
]
},
{
"id": "2e85ceab3abb96f6",
"type": "subflow:3b6a1337b69e1574",
"z": "98b31619.daed3",
"g": "120d335300e6531b",
"name": "",
"x": 4430,
"y": 600,
"wires": [
[
"3dc8c1a1d345ef1d"
]
]
},
{
"id": "f83d15d3a238accd",
"type": "ui-group",
"name": "vvb",
"page": "40ab10feb1ddf2bb",
"width": "30",
"height": "20",
"order": 1,
"showTitle": false,
"className": "",
"visible": "true",
"disabled": "false"
},
{
"id": "40ab10feb1ddf2bb",
"type": "ui-page",
"name": "vvb",
"ui": "cd62f850b848415c",
"path": "/vvb",
"icon": "home",
"layout": "grid",
"theme": "404b2b83a6155d6c",
"breakpoints": [
{
"name": "Default",
"px": "0",
"cols": "30"
},
{
"name": "Tablet",
"px": "1080",
"cols": "30"
},
{
"name": "Small Desktop",
"px": "1080",
"cols": "30"
},
{
"name": "Desktop",
"px": "1080",
"cols": "30"
}
],
"order": 1,
"className": "",
"visible": true,
"disabled": false
},
{
"id": "cd62f850b848415c",
"type": "ui-base",
"name": "flax 24",
"path": "/dashboard",
"includeClientData": true,
"acceptsClientConfig": [
"ui-notification",
"ui-control"
],
"showPathInSidebar": false,
"showPageTitle": false,
"navigationStyle": "default",
"titleBarStyle": "default"
},
{
"id": "404b2b83a6155d6c",
"type": "ui-theme",
"name": "dark jtm",
"colors": {
"surface": "#000000",
"primary": "#0094ce",
"bgPage": "#000000",
"groupBg": "#000000",
"groupOutline": "#000000"
},
"sizes": {
"pagePadding": "12px",
"groupGap": "12px",
"groupBorderRadius": "4px",
"widgetGap": "12px",
"density": "comfortable"
}
}
]
Have you provided an initial effort estimate for this issue?
I am no FlowFuse team member
Is there a way to up-vote this basic request? "a way to have the data in the 2.0 chart node survive restarts"
Further investigation is required on our part on how we can support this. We can't support this natively in the module, as when node-RED restarts, all temporary data is cleared.
The flow shared utilises a file-in/file-out out, which we can investigate, and provide a sample flow for.
I think there will be ongoing requests for this feature. The D1 node accomplishes it by providing an output which can can be saved in file or context and directly re-injected on startup, which is easy for the user to implement and does the job perfectly.
1) Retention of data points within v2.0 (possible to-do flowuse) In my opinion, a perfect basis would be if the ui-graph in version 2.0 would generate an output like in version 1.0. Namely all >displayed< data points (older no more displayed data points are obsolete) as an array, instead of just the last data point received. (The array must actually already be there internally, because how else does the graph display the data points for newly calling clients?)
2) Saving and injecting (to-do for someone else)
There is already a simple external option for permanently saving and then injecting the output array, e.g. with PERSIST.
Alternatively, you can adapt the flow mentioned in this thread.
Yes, I am considering the unified output approach, but we'd also then need to add support for injecting that data format too.
we'd also then need to add support for injecting that data format too.
Could it output data compatible with the existing config of the node, or would that be difficult to achieve?
Hello. Just some positive feedback and also a question that I would assume I will not get an answer to as it is a bit of a "how long is a string" type of question. But here it goes:
We have finished migrating 3 offgrid systems from Dashboard V1 to Dashboard V2 that are now only missing this feature. It has gone much better than expected. D2 seems very stable and robust so far with much better performance than D1. So very happy about that and kudos for great work and the migration tools and docs!
If at all possible. we would like to deploy these systems March-April next year, so is it realistic to hope for this feature by then by any chance? Just planning ahead, and the V1 systems can certainly last for a while more so not vital but would be nice as maintenance for V1 has become a tad cumbersome and stale.
To be clear: A way to have the data in the 2.0 chart node save the data and reload that data / survive restarts.
Vote for this feature. This is currently a big question regarding whether it’s worth transitioning. This feature is mandatory!
With several charts spanning 4 days and even one spanning 6 weeks, persistent chart data during restarte is absolutely vital. I cant upgrade to dash2 on my main system before this is in place...
I build home automation that has a temperature sensor in every room and space, so often 10 or more, each controlling heating via a software thermostat in node red, that in turn writes temperature and power consumption to a chart that is typically 30 days. So the persistent data is vital here too.
Of course, it’s possible to develop a DB solution for this that fetches values every time, but it makes the solution unnecessarily complex. Previously, you could simply use the context store for continuity without a database.
I still advocate for the feature—persistent data is important here as well.
I hope no one is offended by bringing this up again. Just wondering if there has been any development or clever workarounds that I may have missed for " a way to have the data in the 2.0 chart node survive restarts" Many thanks for any comments or updates.
I have written a small subflow which does the job for me for now.
flow.json
[
{
"id": "67eff620cc4aadfd",
"type": "subflow",
"name": "Chart Persistence",
"info": "",
"category": "",
"in": [
{
"x": 80,
"y": 120,
"wires": [
{
"id": "3933c0c996a79eaf"
}
]
}
],
"out": [
{
"x": 1040,
"y": 100,
"wires": [
{
"id": "c92b446ccce9bc9f",
"port": 0
},
{
"id": "f1d4fbf62572f0e1",
"port": 0
}
]
},
{
"x": 1020,
"y": 140,
"wires": [
{
"id": "b6b7ac13ea003979",
"port": 0
}
]
}
],
"env": [
{
"name": "SIZE_NUM",
"type": "num",
"value": "0"
},
{
"name": "SIZE_HRS",
"type": "num",
"value": "0"
}
],
"meta": {},
"color": "#DDAA99",
"inputLabels": [
"single item added to the buffer"
],
"outputLabels": [
"for persist node",
"for chart"
],
"icon": "font-awesome/fa-floppy-o"
},
{
"id": "3933c0c996a79eaf",
"type": "switch",
"z": "67eff620cc4aadfd",
"name": "",
"property": "payload",
"propertyType": "msg",
"rules": [
{
"t": "istype",
"v": "number",
"vt": "number"
},
{
"t": "else"
}
],
"checkall": "true",
"repair": false,
"outputs": 2,
"x": 190,
"y": 120,
"wires": [
[
"f1d4fbf62572f0e1"
],
[
"c93bcb1b9d61130a"
]
]
},
{
"id": "c92b446ccce9bc9f",
"type": "change",
"z": "67eff620cc4aadfd",
"name": "",
"rules": [
{
"t": "set",
"p": "_buffer",
"pt": "flow",
"to": "payload",
"tot": "msg"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 580,
"y": 140,
"wires": [
[
"b6b7ac13ea003979"
]
]
},
{
"id": "f1d4fbf62572f0e1",
"type": "function",
"z": "67eff620cc4aadfd",
"name": "Add data with timestamp and limit array length",
"func": "const now = new Date().getTime();\n\nconst topic = ('topic' in msg) ? msg.topic : '';\n\nvar buffer = flow.get(\"_buffer\");\n\nif ( ! buffer[0].series.includes(topic) )\n{\n buffer[0].series.push(topic);\n buffer[0].data.push([]);\n}\n\nconst idx = buffer[0].series.indexOf(topic);\nbuffer[0].data[idx].push({x: now, y: msg.payload});\n\nconst SIZE_NUM = env.get('SIZE_NUM');\nif ( SIZE_NUM > 0 )\n{\n buffer[0].data[idx].splice(0, buffer[0].data[idx].length - SIZE_NUM);\n}\n\nconst SIZE_HRS = env.get('SIZE_HRS');\nif ( SIZE_HRS > 0 )\n{\n for (const [idx] of buffer[0].series.entries())\n {\n let i = 0;\n while ((i < buffer[0].data[idx].length) && (buffer[0].data[idx][i].x < (now - (SIZE_HRS * 36e5)))) // 1 hour = 60 * 60 * 1000 = 36e5 milliseconds\n {\n i++;\n }\n buffer[0].data[idx].splice(0, i);\n }\n}\n\nflow.set(\"_buffer\", buffer);\n\nreturn {payload: buffer};\n",
"outputs": 1,
"timeout": 0,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 480,
"y": 100,
"wires": [
[
"b6b7ac13ea003979"
]
]
},
{
"id": "b6b7ac13ea003979",
"type": "function",
"z": "67eff620cc4aadfd",
"name": "Generate chart data",
"func": "var chartData = [];\n\nfor (const [idx, topic] of msg.payload[0].series.entries())\n{\n for (const item of msg.payload[0].data[idx])\n {\n chartData.push({topic: topic, ...item});\n }\n}\n\nreturn {payload: chartData};\n",
"outputs": 1,
"timeout": 0,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 820,
"y": 140,
"wires": [
[]
]
},
{
"id": "c93bcb1b9d61130a",
"type": "function",
"z": "67eff620cc4aadfd",
"name": "clean buffer",
"func": "var indices2Delete = [];\n\nfor ( const [idx] of msg.payload[0].series.entries() )\n{\n if ( msg.payload[0].data[idx].length === 0 )\n {\n indices2Delete.push(idx);\n }\n}\n\nfor ( const idx of indices2Delete )\n{\n msg.payload[0].series.splice(idx, 1);\n msg.payload[0].data.splice(idx, 1);\n}\n\nreturn msg;\n",
"outputs": 1,
"timeout": 0,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 370,
"y": 140,
"wires": [
[
"c92b446ccce9bc9f"
]
]
},
{
"id": "eb9c53e4b10f971d",
"type": "inject",
"z": "67eff620cc4aadfd",
"name": "",
"props": [],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": 0.1,
"topic": "",
"x": 110,
"y": 40,
"wires": [
[
"e3f4323dcc3a3f0b"
]
]
},
{
"id": "e3f4323dcc3a3f0b",
"type": "switch",
"z": "67eff620cc4aadfd",
"name": "",
"property": "_buffer",
"propertyType": "flow",
"rules": [
{
"t": "empty"
}
],
"checkall": "true",
"repair": false,
"outputs": 1,
"x": 250,
"y": 40,
"wires": [
[
"bc9ad3179438beb0"
]
]
},
{
"id": "bc9ad3179438beb0",
"type": "change",
"z": "67eff620cc4aadfd",
"name": "",
"rules": [
{
"t": "set",
"p": "_buffer",
"pt": "flow",
"to": "[ {\"series\": [], \"data\": []} ]",
"tot": "json"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 420,
"y": 40,
"wires": [
[]
]
}
]
Unfortunately the persist nodes cannot be part of the subflow, as I could not templatize the storage filename. So you have to add them individually.
I kept the data layout of the original node-red-dashboard 1.0 so the stored data is backwards compatible with dashboard 1.0 charts.
I have written a small subflow which does the job for me for now.
I kept the data layout of the original node-red-dashboard 1.0 so the stored data is backwards compatible with dashboard 1.0 charts.
Thanks @friebi Will try it asap!
Any basic instruction on how to adapt the flow? Wire Input to output of the v2 chart and output to input of the v2 chart? It does not appear to have a config to create a file anywhere. Or does it? Could you briefly elaborate on: "Unfortunately the persist nodes cannot be part of the subflow, as I could not templatize the storage filename. So you have to add them individually."
This looks really interesting – I’m going to try it out.
Still in the long run, Dashboard 2.0 could evolve to include a built-in solution for this – something that works like a native checkbox option. Simple solution.
That would be a real feature upgrade and a step toward making Dashboard 2.0 more advanced.
Any basic instruction on how to adapt the flow? Wire Input to output of the v2 chart and output to input of the v2 chart? It does not appear to have a config to create a file anywhere. Or does it? Could you briefly elaborate on: "Unfortunately the persist nodes cannot be part of the subflow, as I could not templatize the storage filename. So you have to add them individually."
Here is the most simple working example with a V1 and a V2 chart:
Persist nodes: I'm using these ever since: node-red-contrib-persist
This looks really interesting – I’m going to try it out.
Still in the long run, Dashboard 2.0 could evolve to include a built-in solution for this – something that works like a native checkbox option. Simple solution.
That would be a real feature upgrade and a step toward making Dashboard 2.0 more advanced.
I agree. Imho declaring dashboard V1 as deprecated was a bit far fetched. V2 still has quite some obvious glitches and performance issues.
Here is the most simple working example with a V1 and a V2 chart:
I truly appreciate this reply, will give it a go!
Persist nodes: I'm using these ever since: node-red-contrib-persist
Would not persistent context do the job for you?
Imho declaring dashboard V1 as deprecated was a bit far fetched.
It was rapidly becoming impossible to maintain as the framework it was built on reached End of Life a long time ago. It was not initially developed by the node-red core team, but when the developer dropped it, the node-red team took over the maintenance as so many had become reliant on it.
V2 still has quite some obvious glitches and performance issues.
If manpower had been kept aside for maintaining D1 then development of D2 would have been delayed.
Thanks again @friebi !
Trying to understand the flow and not quite working so far. Final Q if my luck with your patience has not run out. No worries if you do not have time, but I am curious to make this work ;) I so far have not seen any debug payloads from the outputs of the "chart persistence" node in proposed subflow. I also assume that when this works as expected, a restart should restore whatever was in the charts before the restart?
- My payloads in my own test example that work when sent directly to the v2 chart and appear to be working and show up in the v2 chart through your subflow are two temp sensors with respective payloads:
upper°C : msg.payload : numberandlower°C : msg.payload : number
I however get an error message from the “clean buffer” function node in your subflow:
TypeError: Cannot read properties of undefined (reading 'entries')
- My best guess is that the function node before the persistence node in your example image does something special with formatting for your subflow and if so can you please share your settings in the persistence nodes or comment?
@colinl thanks for your reply and sorry I was rude. I truly understand your background and we are all doing it on our free time :-) I migrated my full dashboard from V1 to V2 and I can observe a tripled CPU usage of the node-red process on my Raspberry 2 Zero. Also the dashboard page in my browser (esp. on my mobile) is much more laggy than the V1 dashboard with the same content. That is where my complain comes from. Absolute CPU usage is still in an ok range, so I don't have to worry though. I did not yet fully migrate my main instance running on a Raspberry 4.
Would not persistent context do the job for you?
I was reading that for the first time also here in this thread and tbh I wasn't aware of that feature. And I was quickly checking the options in the V1 chart node and can't find this option 🤔 Are you referring to Working with context : Node-RED? Indeed I was not aware of that feature 😆
@houser42 my flow is surely not perfect and only a "works on my machine" implementation 😄 Probably you are passing something else then a pure number into the subflow. Then, the first switch will assume you are passing the full output of the persist node. What is the msg.payload you are providing?
Thanks again @friebi I get it, no worries, but I have been looking for a way to do this for some time ;)
My payload was in my last reply: upper°C : msg.payload : number I need to add topic to payloads to distinguish clearly between sensors in charts.
Are you referring to Working with context : Node-RED?
Yes, I believe that it will do everything that the perist node does, but easier to use. In your subflow you can just save the data to flow context using file system storage. Each subflow has its own flow context so you can do all that inside the subflow.
@colinl Awesome, I did not know this feature so far. I will adapt all my flows. Thank you!
Btw: In order to bring this topic forward, I think a very helpful first step to persistent chart data would be the following:
Currently the chart node returns the msg that was pushed into it. What about adding an additional attribute, let's say _chartData, to the msg which contains all currently visible data points of the chart? That would make my subflow already obsolete and we could just pump that into a change node and assign it to a flow variable. What do you think?
Agree. I believe it’s important to see some progress on this. We need at least the same functionality as in version one—or ideally, an improved solution. As it stands, we can’t roll this out at scale because the data doesn’t persist across reboots. Quite a bit of time has passed, and we may soon need to consider external alternatives. This functionality is fairly critical for our energy-related operations.
Just to jump in, we do have plans to attack this, but also have very limited resource to dedicate at this time.
I would also encourage though, utilising in-memory solutions (as Dashboard 1.0 did and 2.0 would use) does not scale. As soon as you're discussing storage of data cross-restart and for multiple days worth then you should be looking at a database to store that data
utilising in-memory solutions (as Dashboard 1.0 did and 2.0 would use) does not scale
I don't follow you here @joepavitt . At most a chart will hold a couple of thousand samples which should be no problem for file based persistent context storage.
Sure, few thousands, but if you're talking industrial/energy use cases, then that doesn't last you long
You can't show more than a couple of thousand on the chart at any time. We are not talking here about storing historical data, just restoring the real time data that was on the chart before node-red was restarted.
We are not talking here about storing historical data, just restoring the real time data that was on the chart before node-red was restarted
Exactly, perfectly described
As it stands, we can’t roll this out at scale because the data doesn’t persist across reboots. Quite a bit of time has passed, and we may soon need to consider external alternatives. This functionality is fairly critical for our energy-related operations.
If this is important to your organisation you might like to consider paying a consultant with the appropriate skill base to do the work and submit it to the core team for inclusion in the dashboard. This is an open source project so anyone can contribute.