Apply rollout duration of Knative Serving for each InferenceService

[hanwgyu]

/kind feature

Describe the solution you'd like [A clear and concise description of what you want to happen.]

Knative Serving offers rollout duration feature which makes possible to deploy new servers gradually.

It can be applied globally by changing config-network, but i want to specify it for each InferenceService.

As far as i know, it is currently not possible. Is there any workaround for this?

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

[yuzisun]

@hanwgyu you should be able to apply the knative annotation serving.knative.dev/rollout-duration: "380s" on InferenceService as well, that just gets propagated down to the underlying knative service.

[markwinter]

When applying the annotation on InferenceService it gets propagated to spec.template.metadata.annotations in the ksvc.

But serving.knative.dev/rollout-duration is only supported in the top level metadata.annotations of a ksvc unlike other settings for some reason.

Would this be the correct place to set the annotation correctly? https://github.com/kserve/kserve/blob/master/pkg/controller/v1beta1/inferenceservice/reconcilers/knative/ksvc_reconciler.go#L68