Configuration of istio sidecar injection

[strekm]

Description

Istio sidecar injection should be disabled by default. Sidecar injection should be enabled for kyma-system namespace. Implement migration job to label customer namespaces. Update documentation.

Rollout plan:

• [x] reconciler support sidecar injection migration
• [x] note with above plan published for managed kyma
• [x] enable sidecar migration for kyma 2.5.x
• [ ] disable sidecar migration and sidecar injection for kyma 2.6.x
• [ ] adjust Kyma Dashboard UI
• [ ] remove sidecar migration for kyma 2.9.x
• [ ] remove supporting implementation from istio reconciler

Reasons

By default sidecar injection is disabled. We should be as close as possible to Istio default configuration.

Attachments

Part of https://github.com/kyma-project/kyma/issues/11635

[kyma-stale-bot[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[kyma-stale-bot[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[strekm]

With current implementation with sidecar injection enabled by default we need to exclude kube-system from that configuration. Atm we do not have implementation ready to check if reconciliation is needed. With every reconciliation Istio is deployed, under the hood webhook configuration for kube-system ns is wiped out, webhook is patched to exclude kube-system. This leads to a window of time when sidecars are being injected to kube-system. this can lead to possible disruption in the cluster. This situation happen on every reconciliation. Istio implementation of webhooks can differ from version to version. It means that implementation can get quite challenging and error prone. With that mentioned all clusters in multi cluster gardener set up can lost connectivity and this can't be easily brought back up since reconciliation is triggered on a central cluster. see more on here

what we can offer to highlight need of enabling sidecar injection for workload is tutorial explaining advantages of workloads being in istio mesh along with our recommendations. another option would be to offer some kind of tooling analysing pods in given ns and indicating pods w/o sidecar, which then can be visually presented in busola.

[pbochynski]

Istio sidecar injection alone does not introduce any security for the affected workloads/namespaces. The default setting allows accessing the workload from all workloads that are part of a service mesh. Authentication or authorization has to be configured by providing additional istio resources like AuthorizationPolicy(see example https://istio.io/latest/docs/tasks/security/authorization/authz-http/#configure-access-control-for-workloads-using-http-traffic). Therefore I don't see that change affects our security by default principle.

[strekm]

With @pbochynski, @aberezovski and @TorstenD-SAP we decided that we want to do following:

• provide report analysing, which workloads don't have sidecars injected
• provide extensive documentation on how to secure workload (add it to service-mesh and create additional istio resources)

[abbi-gaurav]

Could you please add details about

• How in-cluster eventing would work? Will it work with the default configuration or would require the customer to enable sidecar?
• What would be the impact of the upgrades on such existing setups?

[ptesny]

Having istio ON is very convenient with functions which typically are used to create APIs or web-hooks. What I would suggest is that with new kyma clusters we offer 2 default namespaces: the default one where istio is OFF and for instance a default-with-istio namespace where istion is ON. That way people can easily decide which option (with or w/o istio) they should go on with when deploying their workloads..

[pbochynski]

Another idea: all Kyma controllers that require istio sidecars to work should check if the sidecar injection is enabled and report problems in the status of their resources. Such a solution would give more transparency and will cover more cases than the default sidecar injection enabled on the cluster level. Even with default sidecar injection enabled customers can label their namespaces or deployments (pod template) with sidecar-injection=false. If the event subscription requires sidecar to properly work the eventing should set subscription resource status to failure. The same with APIRule. The status message can suggest that sidecar injection has to be enabled to make use of the functionality. I also do not see any problem with making sidecar injection enabled when you create a new namespace from busola and we can have also some reports or status from istio operator that shows warnings for pods with sidecar injection disabled. In the end, it is more explicit visibility of istio mesh in Kyma cluster than hidden default setting that we rely on and customers are not aware of.