Continue to optimize reconciler performance and reduce latency to take actions on CR events

[terrytangyuan]

Performance optimization for 500 concurrent jobs and large scale completed jobs. Related Issues: https://github.com/kubeflow/tf-operator/issues/965 https://github.com/kubeflow/tf-operator/issues/1079.

This is part of the roadmap.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.98

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[kf-label-bot-dev[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.98

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.