WIP - Bootstrap optimizations - Use cache and use bulk write mongo option to reduce content loading time

[lakshmi-kannan]

closes: https://github.com/StackStorm/st2/issues/4398

[lakshmi-kannan]

Summary of my findings so far:

Bulk write and caching didn't make a meaningful difference to overall bootstrap time for giant packs like AWS which have around 3600 actions.

A single DB write for an action model takes an average of 0.003s. For 3600 records, this takes 1.08 seconds total. By switching to bulk write, we spent almost the same latency (1s average for 3600 bulk actions). There is a point at which the bulk write will make a dent but it won't be much.

The bulk caching at the front also didn't help much. Read single action vs read actions in a pack in bulk was also a wash in terms of numbers comparison. The majority of time is spent on loading JSON from disk and converting it to a DB model after validations.

Here are some example numbers:

2018-11-16 20:52:16,906 INFO [-] [metrics] time(key=st2.register.action.load_content, time=0.01027) 2018-11-16 20:52:16,930 INFO [-] [metrics] time(key=st2.register.action.validate_action_api, time=0.022931) 2018-11-16 20:52:16,936 INFO [-] [metrics] time(key=st2.register.action.validate_action, time=0.005551) 2018-11-16 20:52:16,939 INFO [-] [metrics] time(key=st2.register.action.get_action_db_model, time=0.04314) 2018-11-16 20:52:16,939 INFO [-] [metrics] time(key=st2.register.action.get_db_models.aws, time=94.313884)

So it took 94s to convert all Actions in AWS pack to their corresponding DB models. All operations are CPU bound and are all in memory operations. Converting a single action to DB model took 0.04s which includes JSON load from disk (0.01s), validation against action API model (0.02s), validate parameters with runner params (0.005s).

Next step:

I could look into using an eventlet pool and parallelizing the validation operations. To reduce contention for DB operations, I intend to stick to bulk write but just parallelize DB model conversion operations.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Lakshmi Kannan seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.