Design a generic `interpolation` plugin

[jmcook1186]

Sub of: #656

what

Create a generic interpolation plugin destined to live in if/plugins.

why

We prefer to maintain generic plugins to do arithmetic operations rather than plugins that are hardcoded with parameter values for specific use cases. Specific use cases are then served by combining arithmetic plugins, passing in appropriate values. This makes things much more flexible, configurable and maintainable. This specific plugin is intended to support the deprecation and removal of our teads-curve plugin. Instead of having a plugin just for that specific curve, we can provide a generic interpolation plugin and pass in the known x and y values representing the curve.

Note that to fully replicate the teads-curve plugin the interpolate plugin will have to be part of a pipeline with multiply and divide plugins too.

Acceptance criteria

• [ ] The plugin should support linear, spline and polynomial interpolation methods.

• Linear interpolation is a method of estimating values that fall between known values. It involves determining the value at a point on a straight line between two known points of curve fitting using linear polynomials to construct new data points within the range of a discrete set of known data points.

• Polynomial - Polynomial interpolation involves fitting a polynomial function to a set of data points. We intend to use Lagrange's interpolation. Higher-order polynomials can capture more complex relationships but have over-fitting risks.

• Spline - Spline interpolation uses piecewise-defined polynomial functions (usually low-degree) to interpolate between data points. A spline consists of multiple polynomial segments, often cubic, that are smoothly connected at specific points. Splines provide a balance between flexibility and smoothness, making them suitable for approximating functions with complex variations.

• [ ] The plugin should accept the following global config:

• method - Specifies the interpolation method for the data. Acceptable values are 'linear', 'spline', or 'polynomial'. The default method is linear. (optional)

• x - Array of points. Numbers should be in ascending order (required). (In the old version the points are hardcoded: [0, 10, 50, 100]. Could it be optional to support the same hard-coded values?) (can it be in node-config ?)

• y - Array of numbers. Numbers should be in ascending order (required)

• input-parameter - a string defining the name to use its value to calculate the interpolation. It should match an existing key in the inputs array.

• output-parameter - a string defining the name to use to add the result of interpolation with additional calculation

• [ ] the plugin should validate that the lengths of x and y are equal

• [ ] The plugin should accept the following data from inputs:

• timestamp - a timestamp for the input (required)

• duration - the amount of time, in seconds, that the input covers. (required)

• input-parameter - a field whose name matches the string provided to input-parameter in global config (i.e. if the input-parameter in global config is cpu/utilisation then cpu-utilisation must exist in the input data)

• [ ] The plugin outputs a value z that is the result of looking up the y value at x = input-parameter using the user-defined interpolation method. z is assigned to a field with a name defined by output-parameter in the output data.

• [ ] Unit tests have been added to the builtin repository

• the unit tests accompanying the feature code should have 100% coverage.

• [ ] Documentation has been added in the form of a feature README and linked out from if.greensoftware.foundation

Example Manifest

name: interpolation-demo
description: simple demo of interpolation plugin
tags:
initialize:
 plugins:
   interpolation:
     method: Interpolation
     path: "builtin"
     global-config:
       method: linear
       x: [0, 10, 50, 100]
       y: [0.12, 0.32, 0.75, 1.02]
       input-parameter: “cpu/utilization”
       output-parameter: “interpolation-result”
tree:
 children:
   child:
     pipeline:
       - interpolation
     inputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10

Executing this manifest should yield the following result:

name: interpolation-demo
description: simple demo of interpolation plugin
tags:
initialize:
 plugins:
   interpolation:
     method: Interpolation
     path: "builtin"
     global-config:
       method: linear
       x: [0, 10, 50, 100]
       y: [0.12, 0.32, 0.75, 1.02]
       input-parameter: “cpu/utilization”
       output-parameter: “interpolation-result”
tree:
 children:
   child:
     pipeline:
       - interpolation
     inputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10
    outputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10
         interpolation-result: 0.12

Note on teads curve

The interpolation described in the example above gives the y value from the Tead's curve for any given x value. We pass in cpu/utilisation as an x value and receive the appropriate y. The y value is a scaling factor that should be multiplied by the cpu/thermal-design-power to yield cpu/energy. You may also want to scale the result by the ratio of allocated and total vcpus. Doing all these steps together replicate the existing teads-curve plugin, but only the interpolation part is in scope for this plugin.

[manushak]

Here is the specification doc of the plugin. I've attempted to compose a comprehensive document so anyone who can read it can implement it. @jawache and @jmcook1186 could you please take a look to verify if this is the correct behavior of the plugin?

https://docs.google.com/document/d/1j6VqeB4ibhf0Hp0UxsLL_k8uSD6qXh4eCXzzkEa-ees/edit?usp=sharing

cc: @narekhovhannisyan @MariamKhalatova

[jawache]

Thanks @manushak

I forgot that the teads curve plugin does a lot more than just interpolation 😞

It also multiplies by the TDP and then scales by the VCPUs.

I think we need to solve this by breaking this up into a pipeline like so

• interpolation
• total-cpu-energy (generic multiply by tdp)
• used-cpu-energy (generic divide or multiply by VCPUs avail/used)???

We could call it the "teads-pipeline" or just CPUenergy pipeline.

That's the only way this interpolation plugin is generic enough to justify being a built-in.

If we do the above then the names of the input and output parameters needs to be configurable in global config and you could use interpolation with anything, not just utilisation etc ...

@manushak would the pipeline above work? I have a suspicion they don't work like I wrote above.

[manushak]

@jawache This new interpolation plugin performs the calculation you mentioned: it calculates interpolation using a specified method, multiplies by the cpu/thermal-design-power if provided, then multiplies the result by the duration and divides by 3600 to get seconds in an hour. After that, it divides by 1000 to get kilowatt-hours (kWh). If vcpus-total and vcpus-allocated are provided, the result is multiplied by their division. Does this calculation seem correct to you?

I'm considering making general input and output parameters. I'll update the specification doc and inform you to verify.

[manushak]

@jawache I've updated the spec doc, could you please verify it? (it is very massive, which is why I don't write here)

https://docs.google.com/document/d/1j6VqeB4ibhf0Hp0UxsLL_k8uSD6qXh4eCXzzkEa-ees/edit?usp=sharing

[jmcook1186]

Hi @manushak I just read through the spec docs - this is pretty much what I had in mind for this - thanks!

I'll take your doc and use it to update this issue for you to work against when you are reallocated to IF (unless a contributor gets to it first ;o) ).

It makes sense to me that this plugin only does the interpolation and replicating teads requires some additional steps in a pipeline.

[zanete]

This is ready but will be parked for a while until @manushak can return to this project

[jawache]

@jmcook1186 i also suspect we can't replicate the rest of a teads-like pipeline without some more work on generics (multiply a parameter by the ratio of VCPUs total and reserved)

[jmcook1186]

@manushak please check your PR still conforms to the issue requirements (they changes after your original PR was raised)

[manushak]

@manushak please check your PR still conforms to the issue requirements (they changes after your original PR was raised)

There are small changes to make. Currently, I'm working on it, after that I'll rise a PR