Fix pruning of old data from chart in Append mode #1905

[colinl]

Description

Correct code that prunes old data from the chart. In addition, only prune data if the action is Append. If the action is Replace then I think the users intention will be that all the data added (presumably as a batch of data) should be shown even if it is years old.

Related Issue(s)

#1905

Checklist

• [ x] I have read the contribution guidelines
• [ ] Suitable unit/system level tests have been added and they pass
• [ ] Documentation has been updated
  • [ ] Upgrade instructions
  • [ ] Configuration details
  • [ ] Concepts
• [ ] Changes flowforge.yml?
  • [ ] Issue/PR raised on FlowFuse/helm to update ConfigMap Template
  • [ ] Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production
• [ ] Link to Changelog Entry PR, or note why one is not needed.

Labels

• [ ] Includes a DB migration? -> add the area:migration label

[colinl]

Don't review this yet please, there is another issue https://discourse.nodered.org/t/dashboard-2-0-chart-not-showing-timestamped-data-from-mysql/99412. I think the server side is pruning data on page refresh even in Replace mode.

[colinl]

@Steve-Mcl This is ready for review now. I have added the ability to disable time based pruning by setting the value to 0. This is so that the user can show historical data without it being immediately pruned away.

Does the fact that the docs and help text have been updated need to be flagged to the translators or is there an automatic system for this?

[colinl]

I don't think that the test failures are mine.

[Steve-Mcl]

I don't think that the test failures are mine.

No, they are unrelated (its a odd timing issue we have never got to the bottom of)

[colinl]

It seems that at least one of the chart examples relies on the fact that if historical data are added to the chart then no pruning occurs, unless the page is refreshed. I will fix the example. This does bring up the possibility that users may also be relying on this. I don't know what to do about that, other than note the change in the release notes. The adjustment required is to set the limit time to 0 in the chart.

[colinl]

I have fixed the examples - two of them didn't work even without the latest mods. So this is again ready for review.

[colinl]

The data points come in as an object with properties x and y, but when they are added into the series they go in as and array of two elements, [0] is the x value and [1] is the y value. See this code for example at line 670

                // Add data point
                if (this.props.xAxisType === 'category') {
                    // for categories, we need to update the existing data point for this x-value
                    const xIndex = options.series[sIndex].data.findIndex(d => d[0] === datapoint.x)
                    if (xIndex !== -1) {
                        options.series[sIndex].data[xIndex] = [datapoint.x, datapoint.y]
                    } else {
                        options.series[sIndex].data.push([datapoint.x, datapoint.y])
                    }
                } else {
                    // for other axes, we can just add the data point
                    options.series[sIndex].data.push([datapoint.x, datapoint.y])
                }

[Steve-Mcl]

Merging while acknowledging unrelated e2e test fail.