feat(agent) scale up model with infer delay metric

[lynnmatrix]

What this PR does / why we need it:

1. stats collector works in a general way
2. add support to use model infer delay to scale up model replicas

Stats collector had tight-coupling with lag and last used stats, that was not convenient to add more metric stats, such as model infer delay.

Now, stats collector is defined in a more general way.

Stats collector has a list of ModelStatsKeepers, each ModelStatsKeeper keeps track of one kind of metric(lag, last used, delay) for all models.

ModelReplicaLastUsedKeeper is an implementation of ModelStatsKeeper, it uses LRU cache to record the last used time for each model.

For other metrics(lag, delay), a general ModelStatsKeeper is provided.

General ModelStatsKeeper uses ModelStats to calculate metric for each model.

LagStats and DelayStats implements ModelStats interface.

Which issue(s) this PR fixes: Special notes for your reviewer:

[seldondev]

Hi @lynnmatrix. Thanks for your PR.

I'm waiting for a SeldonIO or todo member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[ukclivecox]

@lynnmatrix Hi - thanks for this. Can you add some more extensive docs for rationale and changes so we can evaluate this more easily?

[lynnmatrix]

@cliveseldon Thanks for your quick response. More description about this PR is added.

[seldondev]

failed to trigger Pull Request pipeline

• failed to create agent
• failed to calculate in repo config
• failed to load trigger config for repository SeldonIO/seldon-core for ref
• failed to switch to ref
• failed to merge repository SeldonIO/seldon-core
• failed to run command in dir /var/tmp/gitrepo1426734524
• git, [merge FETCH_HEAD]
• Updating 8713b15f..c5f150d6 error
• Your local changes to the following files would be overwritten by merge: notebooks/protocol_examples.ipynb operator/controllers/seldondeployment_controller.go operator/controllers/seldondeployment_prepackaged_servers.go operator/controllers/seldondeployment_prepackaged_servers_test.go Please commit your changes or stash them before you merge. Aborting
• exit status 1

[CLAassistant]

All committers have signed the CLA.