influxdata
persist starlark state for more starlark types
Use Case
#15170 added the ability for the starlark processor to persist the global state to the statefile between runs. This is very nice, but it can't store lists, dicts, tuples, or metrics to the state file which severely limits its usefulness. If you try any more complex type you get an error like invalid starlark type *starlark.Metric when it tries to write the state file.
My desired use case is similar to the example for comparing a metric to the previous one. This doesn't work because it stores both a dict and a metric in the global state.
Expected behavior
Starlark standard types (lists, dicts, tuples, and maybe also metrics) can be persisted to the statefile.
Actual behavior
Error when persisting the plugin states.
Additional info
No response
Next steps: implement a function to convert starlark values to go-values and vice versa.
Hi, the statefile saves json, and you can in fact save lists, dicts, strings, numbers, booleans and Nonetype (None, null). (I will call this "simple-type")
To allow for storing arbitrary types could be a problem, because not every object can be stored (Resource handles of any kind (open files, handles for secrets, etc) come to mind).
I would recommend to implement "simple-type"-serializers and deserializers (marshallers and unmarshallers) for every type (or Metric at least) which should be storable.
So have something like
Metric.serialize() -> dict # {"name": ..., "tags": { ... }, "fields": { ... }, "time": ...}
and
Metric.__init__(marsh: dict) -> Metric
Also: state could have an interface which denies everything which isn't a "simple-type", so it would error out before it tries to write it to the statefile.
The metric should be serialized as gob and encoded as base64 to be able to distinct integer/float types which is not possible with pure JSON.
@srebhan, are there any updates on this matter? I also am storing a dictionary of "metrics" in the state of some processors and aggregators.
Upon activating the statefile option my starlark script throws the following error:
state item (-2008395481, Metric("Test Modbus", tags={"bucket": "Energy", "internal_connection": "TestPLC", "internal_deadband": "10.0", "internal_replication": "30s", "internal_timeout": "30s"}, fields={"I1": 800.0}, time=1750796883000000000)) has invalid key type starlark.Int
What can I do to make the metric type stringifyable or do I have to do this writing my own function each time I am storing/deepcopying a metric?
Just gonna leave this example here in case someone wants to store metrics in the state and recreate them after.
def metricToDict(metric):
return {"name": metric.name, "tags": metric.tags, "fields": metric.fields, "time": metric.time}
def dictToMetric(dictMetric):
new_metric = Metric(dictMetric["name"])
new_metric.tags.update(dictMetric["tags"])
new_metric.fields.update(dictMetric["fields"])
new_metric.time = dictMetric["time"]
return new_metric