Encrypted List in Datastore Not Validating as Array When Triggered by IntervalTimer

[Abhi5h3k]

Discussed in https://github.com/StackStorm/st2/discussions/6205

^{Originally posted by Abhi5h3k May 24, 2024}

SUMMARY

I have encountered an unexpected behavior while working with StackStorm workflows involving encrypted data stored in the datastore.

STACKSTORM VERSION

st2 3.8.1, on Python 3.8.10

OS, environment, install method

Docker

Steps to reproduce the problem

Encrypted List Storage: I have stored an encrypted list in the StackStorm datastore using the following command:

st2 key set -e host_list_1 "['host1.com','host2.com']"

My workflow meta:

pack: custom_workflow
enabled: true
runner_type: orquesta
name:  test_usage_workflow
entry_point: workflows/test_usage_workflow.yaml
description: Test usage for the given host to match a threshold
parameters:
  hosts:
    required: true
    type: array
    default: "{{ st2kv.system.host_list_1 | decrypt_kv }}"
    secret: true

Observed Behavior:

The workflow executes successfully when triggered directly from the web UI or using the command line:

st2 run custom_workflow.ram_usage_workflow

However, when the same workflow is triggered using core.st2.IntervalTimer, it fails to initiate and throws a validation error:

jsonschema.exceptions.ValidationError: '{{ st2kv.system.host_list_1 | decrypt_kv}}' is not of type 'array'

Failed validating 'type' in schema['properties']['hosts']:
    {'default': '{{ st2kv.system.host_list_1 | decrypt_kv}}',
     'required': True,
     'secret': True,
     'type': 'array'}

Expected Results

The workflow should correctly decrypt and validate the list stored in the datastore as an array, regardless of whether it is triggered directly or via core.st2.IntervalTimer.

Actual Behavior:

The workflow fails to validate the decrypted list as an array when triggered by core.st2.IntervalTimer.

Request for Assistance:

Could you please help me understand why the validation fails when triggered by core.st2.IntervalTimer and guide me on how to correctly pass a list from the datastore to the default value in my workflow parameters?

[fdrab]

All items in the datastore are stored as strings. Therefore, if you want to fetch something from the datastore that is stored as array, you need to convert it back to an array. You can use | json jinja filter for that (I think, only did this with objects).

EDIT: Or maybe using YAQL would help, I think that one auto converts (not sure tho).

[Abhi5h3k]

@fdrab Thank you for the response.

If the values are stored as JSON-serialized strings, then the data will be automatically parsed into the datatype defined in the parameter definition.

Let's assume this is a issue with string to array conversion. They why does it work without rule timer. Then also it is taking same value from key store.

I have switched to comma separated string for now and also there was a bug with keystore decryption. I have not tested again with array after resolving decryption issue. hopefully it should work.