formio
How to migrate legacy server-side formiojs submission evaluation to new vm strategy?
We are currently in the process of updating our forked formio API from v3 to v4 and have issues understanding how submissions are now evaluated in the new backend (e.g. custom conditions, calculated values etc.).
We have an additional endpoint in our API where we are doing some custom submission evaluations and previously adapted the same strategy used in Validator.validate to achieve this (basically just copying this pipeline: Formio.createForm -> form.checkConditions -> form.calculateValue).
This used to give us the desired "evaluated submission" object which we need to do some additional stuff in our app. But now with v4 the formiojs evaluator got disabled on the serverside and no longer uses the vm environment. Instead the submission evaluation now somehow happens in @formio/core and @formio/vm but it's hard to trace the exact pipeline that's used now.
TL;DR: Is it still possible to create the same "evaluated submission" object previously created by formiojs methods by using the new vm/core methods? Or did the vm refactorings entirely change the way "evaluated submissions" are created and processed in v4 (like the new context.scope object)?
From my current understanding the context.scope object is now the new way of structuring the result of an "evaluated submission" and the old formiojs way of simply merging together the form definition with the submission data does not happen anymore. Is that correct or does the merging still happen, just elsewhere?
We are facing a similar issue while updating from version v3.5.5 to anything equal or after v4.0.0.
We have some calculated hidden fields with custom logic (and "calculateServer": true) and their values are used in some logic from other component validations.
During the submission validation on the server side, these calculations no longer occur. The calculations happen before the values are persisted on DB but not during the validation phase.
Assuming the below scenario.
- A form has a hidden
input_awith a calculated logic. - The same form has an
input_bwith a validation that relies on theinput_a. A malicious user can manipulate the value submitted for theinput_a, and it would behave differently between v3.5.5 and v.4.0.4.
The behavior observed for version v3.5.5:
The input_a value will be recalculated at the server and override the malicious value, resulting in the expected validation of input_b.
The behavior observed for version v4.0.0:
The validation will use the malicious input from input_a to validate the input_b, which is not desirable.
Please let me know if it seems to be the same context as this thread or if I am missing something.
We are facing a similar issue while updating from version v3.5.5 to anything equal or after v4.0.0. We have some calculated hidden fields with custom logic (and
"calculateServer": true) and their values are used in some logic from other component validations. During the submission validation on the server side, these calculations no longer occur. The calculations happen before the values are persisted on DB but not during the validation phase.Assuming the below scenario.
- A form has a hidden
input_awith a calculated logic.- The same form has an
input_bwith a validation that relies on theinput_a. A malicious user can manipulate the value submitted for theinput_a, and it would behave differently between v3.5.5 and v.4.0.4.The behavior observed for version v3.5.5:
The
input_avalue will be recalculated at the server and override the malicious value, resulting in the expected validation ofinput_b.The behavior observed for version v4.0.0:
The validation will use the malicious input from
input_ato validate theinput_b, which is not desirable.Please let me know if it seems to be the same context as this thread or if I am missing something.
As additional information, the order of the components with the calculated value seems to matter in a different way between version 3.5.5 and 4.0.0. For the described scenario, if the input with the calculated value is before the one consuming its result it is not possible to have the input value overridden by the user, and the calculated value will be used as expected.
There should have been a reverse compatibility between 3x and 4x for data processing.
We are interested in any examples which this does not seem to be the case. You are right that we have changed the server-side processing to now use the @formio/core data processors to perform the conditions, calculations, and validations of the server side data, and our goal was to make it so that it is reverse compatible.
What this means is that it SHOULD execute all the scripts that were previously executed in the 3.x but this time it should be done within the isolate-vm (which is WAY more secure than vm2).
Is it possible for you to provide the JSON of a form that works and validates correctly in 3.x but does not work in 4.x? We will take a look on our end of a possible issue in reverse compatibility.
{
"display": "form",
"settings": {
"pdf": {
"id": "1ec0f8ee-6685-5d98-a847-26f67b67d6f0",
"src": "https://files.form.io/pdf/5692b91fd1028f01000407e3/file/1ec0f8ee-6685-5d98-a847-26f67b67d6f0"
}
},
"components": [
{
"collapsible": false,
"key": "panel",
"customConditional": "show=false",
"type": "panel",
"label": "Panel",
"input": false,
"tableView": false,
"components": []
},
{
"collapsible": false,
"key": "panel",
"type": "panel",
"label": "Panel",
"input": false,
"tableView": false,
"components": [
{
"label": "Text Field",
"applyMaskOn": "change",
"tableView": true,
"validateWhenHidden": false,
"key": "textField",
"type": "textfield",
"input": true
}
]
},
{
"type": "button",
"label": "Submit",
"key": "submit",
"disableOnInvalid": true,
"input": true,
"tableView": false
}
]
}
If i have duplicate keys for layout components it gets difficult.
in the old implementation i could ask the form instance for conditional visibility with conditionalVisible() and got the expected results.
Now i can use conditionProcessInfo to calculate visibility and get the info that the component with key panel is hidden ... but i don't know which panel is hidden. Should not be a problem if i only use the form builder as it manages the keys very well but if i have larger forms which in part are only copy paste buildups it is hard to spot this kind of failure. So for the calculation to work, formio should not accept forms with duplicate keys at all ...
Another problem we found are somewhat invalid forms:
{
"display": "form",
"settings": {
"pdf": {
"id": "1ec0f8ee-6685-5d98-a847-26f67b67d6f0",
"src": "https://files.form.io/pdf/5692b91fd1028f01000407e3/file/1ec0f8ee-6685-5d98-a847-26f67b67d6f0"
}
},
"components": [
{
"label": "Day",
"hideInputLabels": false,
"inputsLabelPosition": "top",
"useLocaleSettings": false,
"tableView": false,
"validateWhenHidden": false,
"key": "day",
"type": "day",
"input": true
},
{
"type": "button",
"label": "Submit",
"key": "submit",
"disableOnInvalid": true,
"input": true,
"tableView": false
}
]
}
The missing fields definition on the day component was accepted in the old implementation but in the new version the validation throws an error. Same idea here, formio should not accept this form in the first place, or be more tolerant to missing fields as the old implementation.
for the first issue i filled a separate bug at https://github.com/formio/formio/issues/1819