st2 icon indicating copy to clipboard operation
st2 copied to clipboard

Published 20 hours ago • verified publisher icon StackStorm

Rendering rule action parameters with quoted strings causes an error

Open nmaludy opened this issue 6 years ago • 5 comments

SUMMARY

When invoking a rule that processes a trigger with a value containing a quoted string causes an error.

ISSUE TYPE
  • Bug Report
STACKSTORM VERSION
# st2 --version
st2 3.0.0, on Python 2.7.5
OS / ENVIRONMENT / INSTALL METHOD
CentOS 7.6
Puppet-st2
STEPS TO REPRODUCE

rule.yaml

---
name: "quoted_param_in_trigger_bug"
pack: "default"
description: "Demos how quoted values in a trigger instance cause render failures in a rule."
enabled: true

trigger:
  type: "core.st2.key_value_pair.update"

criteria: {}
action:
  ref: "examples.test"
  parameters:
    data:
      test: "{{ trigger.object.value }}"

trigger.yaml (yes, this file needs to be called .yaml even though it has JSON data in it)

{
    "id": "5ceed2bb9387ef0cefd1384b", 
    "occurrence_time": "2019-05-29T14:43:07.000000Z", 
    "payload": {
        "object": {
            "encrypted": false, 
            "id": "5ceed29d9387ef2a25148f26", 
            "name": "nick_test", 
            "scope": "st2kv.system", 
            "secret": false, 
            "uid": "key_value_pair:st2kv.system:nick_test", 
            "value": "Nick Test \"quoted string data\" blah"
        }
    }, 
    "status": "processed", 
    "trigger": "core.st2.key_value_pair.update"
}
EXPECTED RESULTS
$ st2-rule-tester --rule=./rule.yaml --trigger-instance=./trigger.yaml --config-file=/etc/st2/st2.conf
2019-05-29 14:49:24,439 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:49:24,443 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:49:24,592 INFO [-] Validating rule default.quoted_param_in_trigger_bug for st2.key_value_pair.update.
2019-05-29 14:49:24,592 INFO [-] 1 rule(s) found to enforce for st2.key_value_pair.update.
2019-05-29 14:49:24,593 INFO [-] Failed to retrieve config for pack <Mock name='mock.pack' id='139681899526160'> and user stanley: 'Mock' object is not iterable
2019-05-29 14:49:24,598 INFO [-] Action parameters resolved to:
2019-05-29 14:49:24,598 INFO [-]        data: {u'test': u'Nick Test "quoted string data" blah'}
2019-05-29 14:49:24,598 INFO [-] === RULE MATCHES ===
ACTUAL RESULTS

Note: i have removed a bunch of the exception catching code so we can see the real stack trace instead of the "masked" version that hides the root of the issue.

$ st2-rule-tester --rule=./rule.yaml --trigger-instance=./trigger.yaml --config-file=/etc/st2/st2.conf
2019-05-29 14:48:54,608 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:48:54,612 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:48:54,752 INFO [-] Validating rule default.quoted_param_in_trigger_bug for st2.key_value_pair.update.
2019-05-29 14:48:54,752 INFO [-] 1 rule(s) found to enforce for st2.key_value_pair.update.
2019-05-29 14:48:54,753 INFO [-] Failed to retrieve config for pack <Mock name='mock.pack' id='140023090276368'> and user stanley: 'Mock' object is not iterable
Traceback (most recent call last):
  File "/usr/bin/st2-rule-tester", line 21, in <module>
    rule_tester.main()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/cmd/rule_tester.py", line 58, in main
    matches = tester.evaluate()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/rules/tester.py", line 106, in evaluate
    additional_contexts=additional_contexts)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/rules/enforcer.py", line 84, in get_resolved_parameters
    additional_contexts=additional_contexts)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 312, in render_live_params
    context = _resolve_dependencies(G)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 224, in _resolve_dependencies
    context[name] = _render(node, context)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 208, in _render
    result = json.loads(result)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 1 column 22 (char 21)

nmaludy avatar May 29 '19 18:05 nmaludy

Debugging

If i run with --debug you can see the render parameter issue:

$ st2-rule-tester --rule=./rule.yaml --trigger-instance=./trigger.yaml --config-file=/etc/st2/st2.conf --debug
2019-05-29 14:54:15,053 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:54:15,058 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-05-29 14:54:15,058 DEBUG [-] Ensuring database indexes...
2019-05-29 14:54:15,152 DEBUG [-] Skipping index cleanup for blacklisted model "PermissionGrantDB"...
2019-05-29 14:54:15,197 DEBUG [-] Indexes are ensured for models: ActionAliasDB, ActionAliasDB, ActionDB, ActionExecutionDB, ActionExecutionDB, ActionExecutionOutputDB, ActionExecutionSchedulingQueueItemDB, ActionExecutionStateDB, ActionExecutionStateDB, ApiKeyDB, ConfigDB, ConfigSchemaDB, GroupToRoleMappingDB, KeyValuePairDB, LiveActionDB, LiveActionDB, PackDB, PermissionGrantDB, PolicyDB, PolicyTypeDB, RoleDB, RuleDB, RuleEnforcementDB, RunnerTypeDB, RunnerTypeDB, SensorTypeDB, TaskExecutionDB, TokenDB, TraceDB, TriggerDB, TriggerInstanceDB, TriggerTypeDB, UserDB, UserRoleAssignmentDB, WorkflowExecutionDB
2019-05-29 14:54:15,204 INFO [-] Validating rule default.quoted_param_in_trigger_bug for st2.key_value_pair.update.
2019-05-29 14:54:15,204 DEBUG [-] Trigger payload: {u'object': {u'uid': u'key_value_pair:st2kv.system:nick_test', u'encrypted': False, u'value': u'Nick Test "quoted string data" blah', u'secret': False, u'scope': u'st2kv.system', u'id': u'5ceed29d9387ef2a25148f26', u'name': u'nick_test'}}
2019-05-29 14:54:15,204 DEBUG [-] [1st_pass] 1 rule(s) found to enforce for st2.key_value_pair.update.
2019-05-29 14:54:15,204 DEBUG [-] [2nd_pass] 0 rule(s) found to enforce for st2.key_value_pair.update.
2019-05-29 14:54:15,204 INFO [-] 1 rule(s) found to enforce for st2.key_value_pair.update.
2019-05-29 14:54:15,205 DEBUG [-] Attempting to get config for pack "<Mock name='mock.pack' id='139946342672592'>" and user "stanley"
2019-05-29 14:54:15,205 DEBUG [-] Pack and user found. Loading config.
2019-05-29 14:54:15,205 INFO [-] Failed to retrieve config for pack <Mock name='mock.pack' id='139946342672592'> and user stanley: 'Mock' object is not iterable
2019-05-29 14:54:15,208 DEBUG [-] Template ast: Template(body=[Output(nodes=[TemplateData(data=u"{'test': '"), Getattr(node=Getattr(node=Name(name='trigger', ctx='load'), attr='object', ctx='load'), attr='value', ctx='load'), TemplateData(data=u"'}")])])
2019-05-29 14:54:15,209 DEBUG [-] Dependencies: set(['trigger'])
2019-05-29 14:54:15,209 DEBUG [-] Rendering complex type: {"test": "{{ trigger.object.value }}"}
2019-05-29 14:54:15,209 DEBUG [-] Rendering node: {'template': '{"test": "{{ trigger.object.value }}"}'} with context: {'trigger': {u'object': {u'uid': u'key_value_pair:st2kv.system:nick_test', u'encrypted': False, u'value': u'Nick Test "quoted string data" blah', u'secret': False, u'scope': u'st2kv.system', u'id': u'5ceed29d9387ef2a25148f26', u'name': u'nick_test'}}, 'action_context': {'trigger_instance': {'id': 'rule_tester_instance', 'name': None}, 'user': 'stanley', 'rule': {'id': 'rule_tester_rule', 'name': u'quoted_param_in_trigger_bug'}, 'pack': <Mock name='mock.pack' id='139946342672592'>}, 'st2kv': {'system': <st2common.services.keyvalues.KeyValueLookup object at 0x7f47cc0ae850>, 'user': <st2common.services.keyvalues.UserKeyValueLookup object at 0x7f47cc0aeb10>}}
2019-05-29 14:54:15,210 DEBUG [-] Render complete: {"test": "Nick Test "quoted string data" blah"}
Traceback (most recent call last):
  File "/usr/bin/st2-rule-tester", line 21, in <module>
    rule_tester.main()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/cmd/rule_tester.py", line 58, in main
    matches = tester.evaluate()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/rules/tester.py", line 106, in evaluate
    additional_contexts=additional_contexts)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/rules/enforcer.py", line 84, in get_resolved_parameters
    additional_contexts=additional_contexts)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 312, in render_live_params
    context = _resolve_dependencies(G)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 224, in _resolve_dependencies
    context[name] = _render(node, context)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 208, in _render
    result = json.loads(result)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 1 column 22 (char 21)

The final line:

2019-05-29 14:54:15,210 DEBUG [-] Render complete: {"test": "Nick Test "quoted string data" blah"}

You can see the quotes inside the string are not escaped resulting in invalid JSON

nmaludy avatar May 29 '19 18:05 nmaludy

FYI i've been able to work around this problem using the | json_escape filter in my rule:

---
name: "quoted_param_in_trigger_bug"
pack: "default"
description: "Demos how quoted values in a trigger instance cause render failures in a rule."
enabled: true

trigger:
  type: "core.st2.key_value_pair.update"

criteria: {}
action:
  ref: "examples.test"
  parameters:
    data:
      test: "{{ trigger.object.value | json_escape }}"

nmaludy avatar May 29 '19 19:05 nmaludy

The | json_escape trick only works if you're extracting a single field. If you're extracting an object or array you still get an error, see the rule below:

---
name: "quoted_param_in_trigger_bug"
pack: "default"
description: "Demos how quoted values in a trigger instance cause render failures in a rule."
enabled: true

trigger:
  type: "core.st2.key_value_pair.update"

criteria: {}
action:
  ref: "examples.test"
  parameters:
    data:
      test: "{{ trigger.object | json_escape }}"

Since i'm getting the entire trigger.object it fails to render. If i limit the data to extract just the single field trigger.object.value then | json_escape properly escapes the string and can work around the error.

nmaludy avatar May 29 '19 19:05 nmaludy

Thanks for making this ticket @nmaludy

punkrokk avatar May 29 '19 22:05 punkrokk

Failed to retrieve config for pack <Mock name='mock.pack' id='140714291121680'> and user digitalxcadmin: expected string or bytes-like object i am getting error like this give me a sollution for this.

PLPRASU avatar May 09 '22 00:05 PLPRASU