Migration: Chart Input/Output Formats

[MarianRaphael]

One issue during the migration of one of my dashboards was that the output and expected input from Dashboard 2.0's chart differ. Previously, it was possible to store the output of the chart and insert the stored values upon instance refresh. This is now different, which makes it hard to create a persistent dashboard. An easier migration would be possible if the old chart input format were also accepted.

See: https://flowfuse.com/blog/2023/05/persisting-chart-data-in-node-red/

Requirements

• [ ] Dashboard 2.0 should accept the old chart input format alongside the new one.
• [x] Support for Year-Month-Date on X-axis
• [x] Output of the chart node should be compatible with input

[joepavitt]

Not sure I know the problem here regarding input? Can you be more specific please?

They should be identical?

[joepavitt]

Output of the chart node should be compatible with input

Just checked this:

Output is identical to input:

[joepavitt]

Support for Year-Month-Date on X-axis

This is also supported already:

[MarianRaphael]

Here is the example, a new dashboard with old dashboard data: https://product-metrics-dev.flowforge.cloud/?#flow/927154a090f3f116

[Steve-Mcl]

This might help Joe: https://github.com/node-red/node-red-dashboard/blob/master/Charts.md

[joepavitt]

Here is the example, a new dashboard with old dashboard data:

Can you send me an example of the output from "Old Dashboard Data" please? I'm not able to trigger that myself, nor is there anything in the debug console.

[joepavitt]

Having investigated this further, I can see that you're actually passing the output from the Dashboard 1.0 ui_chart to the input of the Dashboard 2.0 chart. We do not currently support that data format (with series, data, labels, etc.)

If, instead, you were to pass the same input to both the D1.0 and D2.0 chart, then it would work.

[joepavitt]

I need to spend some time understanding what D1.0 charts output, as it's inconsistent with the D1.0 input. It would need to be a new feature implemented in order to support the format:

[{
"series": ["A", "B", "C"],
"data": [
    [{ "x": 1504029632890, "y": 5 },
     { "x": 1504029636001, "y": 4 },
     { "x": 1504029638656, "y": 2 }
    ],
    [{ "x": 1504029633514, "y": 6 },
     { "x": 1504029636622, "y": 7 },
     { "x": 1504029639539, "y": 6 }
    ],
    [{ "x": 1504029634400, "y": 7 },
     { "x": 1504029637959, "y": 7 },
     { "x": 1504029640317, "y": 7 }
    ]
],
"labels": [""]
}]

I also don't have clarity on when or how that data is extracted from an existing chart in D1.0, how have you done with @MarianRaphael? It doesn't run for each new data point injection, appears to only be on a redeploy, or maybe a page view for a chart? It's not documented, or clear as far as I can tell.

[Steve-Mcl]

Hey Joe/Marian, I might be barking up the wrong neck of the woods but the output of the chart should not be a consideration (IMO) only the input. I have never seem (or used) the D1 chart output. I have many times written flows that take users input (i.e. buttons, texts, datetime entries), pushed these to a SQL query and fed the DB results through a FN or change node (to coerce the data to the above format) and then sent data to the chart node. I feel this is the most important part to support.

And if you feel the input formats of D1 are unwieldy or not ideally suited for latest Chart.js version, I believe so long as we can push an array of data objects to the chart (say from a database) and it populates the chart, that is enough for a start.

Other missing features like what the chart outputs or changing chart props/colours/line endings etc can come later.

[joepavitt]

I might be barking up the wrong neck of the woods but the output of the chart should not be a consideration (IMO) only the input.

I agree.

I feel this is the most important part to support

I agree

I believe so long as we can push an array of data objects to the chart (say from a database) and it populates the chart, that is enough for a start.

We already support this: https://dashboard.flowfuse.com/nodes/widgets/ui-chart.html#multiple-data-points-injecting-an-array-of-data

[MarianRaphael]

The D1.0 Dashboard is triggered once a day and calculates several product metrics. Please don't hesitate to trigger the "Load the data from context on deploy" whenever needed: https://product-metrics.flowforge.cloud/#flow/a65a1ad66f55a68b

I believe so long as we can push an array of data objects to the chart (say from a database) and it populates the chart, that is enough for a start.

But this is exactly what I do with persistent storage instead of a database. What I can change is to capture the input instead of the output of the chart node. I just wanted to show that there is a gap between the behaviours. Rob and I built all our dashboards this way, but I think the change should be quickly realized.

[joepavitt]

But this is exactly what I do with persistent storage instead of a database. What I can change is to capture the input instead of the output of the chart node

The difference is (currently) in the format that multiple data points can be added to a chart from D1.0 to D2.0:

D1.0:

[{
   "series": ["A", "B", "C"],
   "data": [
      [
         { "x": 1504029632890, "y": 5 },
         { "x": 1504029636001, "y": 4 },
         { "x": 1504029638656, "y": 2 }
      ],
      [
         { "x": 1504029633514, "y": 6 },
         { "x": 1504029636622, "y": 7 },
         { "x": 1504029639539, "y": 6 }
      ],
      [
         { "x": 1504029634400, "y": 7 },
         { "x": 1504029637959, "y": 7 },
         { "x": 1504029640317, "y": 7 }
      ]
   ],
   "labels": [""]
}]

D2.0: Note limitation here with my new design is one line per msg. I am happy to implement the D1.0 way of loading past data too.

msg = {
    "topic": "line-1",
    "payload": [{
        x: 30,
        y: 43
    }, {
        x: 40,
        y: 56
    }, {
        x: 50,
        y: 74
    }]
}

[MarianRaphael]

See also: https://flowfuse.com/blog/2023/05/persisting-chart-data-in-node-red/

[joepavitt]

Thanks Marian, wasn't even aware that was something we did! We should be utilising that in the OKR Dashboard

[pfjarschel]

I am new to this package, but why not support something more simple, such as :

msg = {
    "topic": "line-1",
    "payload": 
        x: [0, 1, 2, 3, 4],
        y: [1, 4, 7, 3, 8]
}

?

Makes way more sense to me.

[joepavitt]

Appreciate the feedback @pfjarschel - generally the data sources we find users are working with are coming in on a message-by-message basis (e.g. reading from live instrumentation such as {time: <time>, temp: <temp>}), or from a REST API which generally returns an array of data with unique data points like [{time: <time>, temp: <temp>}, {time: <time>, temp: <temp>}], so they're the ones we've prioritised supporting for the moment.

Not adverse to supporting that format at all though! The main complexity with the proposed format there is missing data, if one array is a different length to the other, do we just assume that the missing data point is the last one, and pair up the rest? Ignore entirely?

[pfjarschel]

Appreciate the response!

The way matplotlib, the most used python plotting and charting package, handles this, is throwing an exception if the arrays are of different length. I do not believe this is a good approach though, but something in this line would be acceptable, I believe. Or cropping the largest array to the size of the smallest, and giving a warning of some kind.

But anyway, thank you for your reply! I think my use case (getting time and voltage data from an oscilloscope, for example) is very different from most people that uses this package and js in general, that's why it struck me as odd, but it makes sense from a web application point of view.

[Cruzxia]

I have just started migrating to dashboard 2.0 and I have run into the chart data format issue.

Would it be possible to add a data type selection option in the chart setup to allow the chart data to be in dashboard 1.0 formats. I have years of chart data, so it would be more efficient if it could be plotted directly to the chart, without having to write a data processing loop to convert it to the revised format.