StackStorm
zstandard compression for parameters and results
why
mitigate (as much as possible) 'document too large' mongo exceptions. 16 mb is the limit for mongodb document size.
done
- zstandard compression for parameter and results
- remove embedded liveaction doc in executions db and replace with liveaction id string. This change will potentially save 1/2 the time to write the liveaction to the database as the liveaction won't be written 2 times.
- added traceback info to exception handling for action execution engine and workflow engine. This should make it easier to debug for someone when the workflow fails for some reason. invalid orjson or unterminated yaql strings cause these exceptions in my experience.
- This will require a data migration for the
executionDBmodel for theliveactionparamter as it is now a string as opposed to an embedded document - I added a config option to turn off zstandard compression for parameters and results in the execution db and the liveaction db
@amanda11 @cognifloyd would you please take a look at this and give me some feedback. I am interested in if the direction the code is going in is acceptable.
We are having numerous instances now of mongo documents being too large. I ran a check and one of the larger objects we have would compress down to .5 mb from 10 mb using zstandard.
Benchmark results
TLDR. zstandard is much faster for single field large json zstandard is about the same time for all other tests. +/- a few ticks
The cost here is minimal from a speed perspective
$ /st2/st2common/benchmarks/micro$ pytest test_mongo_field_types.py --benchmark-columns=mean,rounds
--------------------------------- benchmark 'live_action_read': 12 tests --------------------------------
Name (time in ms) Mean Rounds
---------------------------------------------------------------------------------------------------------
test_read_large_execution[old_json_dict_field-tiny_1] 2.3494 (1.0) 387
test_read_large_execution[json_dict_field-tiny_1] 2.3697 (1.01) 391
test_read_large_execution[old_json_dict_field-json_61kb] 3.2883 (1.40) 254
test_read_large_execution[json_dict_field-json_61kb] 3.3988 (1.45) 269
test_read_large_execution[old_json_dict_field-json_4mb_single_large_field] 17.2653 (7.35) 56
test_read_large_execution[json_dict_field-json_4mb_single_large_field] 12.3281 (5.25) 78
test_read_large_execution[old_json_dict_field-json_647kb] 13.6826 (5.82) 66
test_read_large_execution[json_dict_field-json_647kb] 15.9034 (6.77) 67
test_read_large_execution[old_json_dict_field-json_4mb] 114.0750 (48.55) 11
test_read_large_execution[json_dict_field-json_4mb] 115.1644 (49.02) 11
test_read_large_execution[old_json_dict_field-json_8mb] 246.7735 (105.03) 5
test_read_large_execution[json_dict_field-json_8mb] 248.3191 (105.69) 5
---------------------------------------------------------------------------------------------------------
--------------------------------- benchmark 'live_action_save': 12 tests --------------------------------
Name (time in ms) Mean Rounds
---------------------------------------------------------------------------------------------------------
test_save_large_execution[old_json_dict_field-tiny_1] 6.5457 (1.0) 25
test_save_large_execution[json_dict_field-tiny_1] 6.7954 (1.04) 134
test_save_large_execution[old_json_dict_field-json_61kb] 11.6969 (1.79) 78
test_save_large_execution[json_dict_field-json_61kb] 13.1218 (2.00) 75
test_save_large_execution[old_json_dict_field-json_4mb_single_large_field] 30.7534 (4.70) 28
test_save_large_execution[json_dict_field-json_4mb_single_large_field] 18.3983 (2.81) 54
test_save_large_execution[old_json_dict_field-json_647kb] 73.0526 (11.16) 14
test_save_large_execution[json_dict_field-json_647kb] 81.4653 (12.45) 13
test_save_large_execution[old_json_dict_field-json_4mb] 381.4303 (58.27) 5
test_save_large_execution[json_dict_field-json_4mb] 385.8807 (58.95) 5
test_save_large_execution[old_json_dict_field-json_8mb] 737.6227 (112.69) 5
test_save_large_execution[json_dict_field-json_8mb] 756.7961 (115.62) 5
---------------------------------------------------------------------------------------------------------
-------------------------- benchmark 'live_action_save_multiple_fields': 10 tests -------------------------
Name (time in ms) Mean Rounds
-----------------------------------------------------------------------------------------------------------
test_save_multiple_fields[old_json_dict_field-tiny_1] 6.6645 (1.0) 113
test_save_multiple_fields[json_dict_field-tiny_1] 6.7131 (1.01) 123
test_save_multiple_fields[old_json_dict_field-json_61kb] 22.1928 (3.33) 47
test_save_multiple_fields[json_dict_field-json_61kb] 24.7535 (3.71) 41
test_save_multiple_fields[old_json_dict_field-json_4mb_single_large_field] 119.9479 (18.00) 10
test_save_multiple_fields[json_dict_field-json_4mb_single_large_field] 37.2357 (5.59) 25
test_save_multiple_fields[old_json_dict_field-json_647kb] 198.9707 (29.86) 5
test_save_multiple_fields[json_dict_field-json_647kb] 228.3664 (34.27) 5
test_save_multiple_fields[old_json_dict_field-json_4mb] 1,144.1683 (171.68) 5
test_save_multiple_fields[json_dict_field-json_4mb] 1,139.4643 (170.98) 5
-----------------------------------------------------------------------------------------------------------
This will require a data migration for the executionDB model for the liveaction paramter as it is now a string as opposed to an embedded document
data migration tested
$ ./st2-migrate-liveaction-executiondb --config-file ../../../../conf/st2.dev.conf
StackStorm v3.9 database field data migration script
Will migrate objects with creation date between 2023-06-06 13:01:20 UTC and 2023-07-06 13:01:20 UTC.
You are strongly recommended to create database backup before proceeding.
Depending on the number of the objects in the database, migration may take multiple hours or more. You are recommended to start the script in a screen session, tmux or similar.
To proceed with the migration, press enter and to cancel it, press CTRL+C.
Migrating affected database objects between 2023-06-06 13:01:20 and 2023-07-06 13:01:20
Migrating execution objects
Will migrate 10 ActionExecutionDB objects
[1/10] Migrating ActionExecutionDB with id 64a6ba4fdce54d96db28f4ae
ActionExecutionDB with id 64a6ba4fdce54d96db28f4ae has been migrated
[2/10] Migrating ActionExecutionDB with id 64a6ba40dce54d96db28f4a2
ActionExecutionDB with id 64a6ba40dce54d96db28f4a2 has been migrated
[3/10] Migrating ActionExecutionDB with id 64a6ba3ddce54d96db28f49c
ActionExecutionDB with id 64a6ba3ddce54d96db28f49c has been migrated
[4/10] Migrating ActionExecutionDB with id 64a6ba4edce54d96db28f4ab
ActionExecutionDB with id 64a6ba4edce54d96db28f4ab has been migrated
[5/10] Migrating ActionExecutionDB with id 64a6ba50dce54d96db28f4b7
ActionExecutionDB with id 64a6ba50dce54d96db28f4b7 has been migrated
[6/10] Migrating ActionExecutionDB with id 64a6ba3fdce54d96db28f49f
ActionExecutionDB with id 64a6ba3fdce54d96db28f49f has been migrated
[7/10] Migrating ActionExecutionDB with id 64a6ba4fdce54d96db28f4b1
ActionExecutionDB with id 64a6ba4fdce54d96db28f4b1 has been migrated
[8/10] Migrating ActionExecutionDB with id 64a6ba4edce54d96db28f4a8
ActionExecutionDB with id 64a6ba4edce54d96db28f4a8 has been migrated
[9/10] Migrating ActionExecutionDB with id 64a6ba41dce54d96db28f4a5
ActionExecutionDB with id 64a6ba41dce54d96db28f4a5 has been migrated
[10/10] Migrating ActionExecutionDB with id 64a6ba50dce54d96db28f4b4
ActionExecutionDB with id 64a6ba50dce54d96db28f4b4 has been migrated
SUCCESS: All database objects migrated successfully (duration: 0 seconds).
I added a config option to turn off zstandard compression for parameters and results in the execution db and the liveaction db
Since liveaction field of the ActionExecutionDB was required before I made liveaction_id also required. This should help ensure the liveaction actually exists.
also, what is up with the circle-ci failures? is there anything we can do to fix that?
For the migration, I wonder if there's a raw query we can use that delegates the migration to mongo. Something like
{"$set": {"liveaction_id": doc.liveaction.id}}
I changed the migration to use a raw query. I couldn't get it to work in one update statement so I had to write a for loop.
@cognifloyd The circle ci is fixed. rabbitmq needs priv=true and redis needed to be pinned. pr. can you merge?
I think zstandard compression is a good change in case numbers across various types of datasets confirm it's indeed worth it / good trade off of CPU cycles used for compressing / decompressing those field values.
We decided to not enable compression by default as part of #4846 to keep things simple and reduce the scope of those changes. I'm fine with enabling the compression by default now, in case it numbers still back this up.
As far as this change goes:
remove embedded liveaction doc in executions db and replace with liveaction id string. This change will potentially save 1/2 the time to write the liveaction to the database as the liveaction won't be written 2 times.
This one I'm not too sure about it yet and I need to dig in and think about it some more.
I think it would be good if we split this PR into two - one for field compression changes and one for field value / foreign key change.
In the end, both changes are somewhat related, but they have vastly different implications - one was already benchmarked in the past, it's backward / forward compatible and the other one is not, it could have large implications and also requires a potentially expensive and breaking migration (which is not backward compatible).
Thanks @kami for the review. I would be ok just leaving this a breaking change. I don't have a lot of cycles to split it up. Thanks.
FWIW we have forked over to this branch along with removing the deep copies in orquesta . We are seeing huge speed improvements. Also, no more Mongo document too large.