[Security Pals] Flux multi-tenancy proposal

[pjbgf]

Project Name: Flux

Github URL: https://github.com/fluxcd Key sub-projects:

• https://github.com/fluxcd/flux2
• https://github.com/fluxcd/source-controller
• https://github.com/fluxcd/helm-controller
• https://github.com/fluxcd/kustomize-controller
• https://github.com/fluxcd/notification-controller
• https://github.com/fluxcd/image-automation-controller
• https://github.com/fluxcd/image-reflector-controller
• https://github.com/fluxcd/pkg
• https://github.com/fluxcd/flagger

CNCF project stage and issue (NA if not applicable): in-flight proposal for Graduation

• Sandbox Proposal: https://github.com/cncf/toc/pull/232
• Incubation Proposal: https://github.com/cncf/toc/pull/567
• Graduation Proposal (in-flight): https://github.com/cncf/toc/pull/796

Security Provider: No

• [ ] Identify team
  • [x] Project security lead - @pjbgf
  • [ ] Lead security reviewer
  • [ ] 1 or more additional reviewer(s)
  • [ ] Every reviewer has read security reviewer guidelines and stated declaration of conflict
  • [ ] Sign off by 2 chairs on reviewer conflicts
• [ ] Create slack channel (e.g. #sec-assess-projectname)
• [ ] Project lead provides draft document - see outline
• [ ] "Naive question phase" Lead Security Reviewer asks clarifying questions
• [ ] Assign issue to security reviewers
• [ ] Initial review
• [ ] Presentation & discussion
• [ ] Share draft findings with project
• [ ] Assessment summary and doc checked into /assessments/projects/project-name (require at least 1 co-chair approval)
• [ ] CNCF TOC presentation (if requested by TOC)

Self-assessment: Multi-tenancy: https://docs.google.com/document/d/1SluYVDuq-egSTurcnrVRMJw6ecSV65Qtgi10T4WHyYs/edit# Flagger: https://docs.google.com/document/d/1bdsWHT1L403ss1meMF6zR1G4hUy2qLtIi8I-IMKEMmM/edit

Context:

Flux had its first security audit in November 2021. Multi-tenancy was mostly out of scope, however the report advised us to engage "with experts, such as the CNCF Security Technical Advisory Group, on both the design of the underlying user system and also on the implementation of the security model."

Therefore this assessent scope will focus on the current and proposed changes of Flux in multi-tenancy environments.

---

EDIT: Added flagger details and self-assessment.

[lumjjb]

@IAXES

[lumjjb]

TODO: @cncf/tag-security need to discuss with TOC @TheFoxAtWork @justincormack on next steps.

[lumjjb]

Notes from our meeting to discuss next steps:

• [@pjbgf/Hidde] Will be helpful to have a matrix of resource vs deployment model matrix (table visualization of threat model)
• [@pjbgf/Hidde] Create a presentation issue with what dates work and we will schedule a time to present multi-tenancy, to find someone to try and help more closely with this.
• [@achetal01 ] is going to dive a bit more on it and comment on the proposal
• [@lumjjb] Will communicate with the TOC on planned next steps

• Flux will add an additional matrix to help better interpret the threat model against the various types of deployment models available
• Flux multi-tenancy proposal will be presented at TAG to drive the security pals process.
• k8s multi-tenancy WG will release some definitions around soft/hard multitenancy. In the future, flux will write something up around the Flux deployment model compared with the WG definitions.

[TheFoxAtWork]

Requested access to the multi tenancy doc

[pjbgf]

Following-up from our meeting back in May, I have updated the Self-Assessment documents with further information about Multi-Tenancy models (inc. an initial thread model) and the Flux Security Best Practices (for users).

We have also submitted an issue to present Multi-tenancy. Please let us know whether anything else is required ahead of the presentation.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[pjbgf]

I was away for a couple of weeks and just got back now. Is there anything that the Flux team can do to help on progressing this issue?

[achetal01]

Paulo

I had reviewed the threat model and provided some comments on additional mitigations .Not sure you received those.

Aradhna

On Wed, Sep 28, 2022 at 2:09 AM Paulo Gomes @.***> wrote:

I was away for a couple of weeks and just got back now. Is there anything that the Flux team can do to help on progressing this issue?

— Reply to this email directly, view it on GitHub https://github.com/cncf/tag-security/issues/896#issuecomment-1260613698, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARO764QN6JHLSCOMHSLWRBDWAQDL3ANCNFSM5UKX2W7A . You are receiving this because you were mentioned.Message ID: @.***>

[sublimino]

@pjbgf @stefanprodan are either of you and/or any of the team free at 1300 BST tomorrow for the EMEA TAG Security meeting? We have space to run through an initial threat modelling session for this proposal with our new lightweight framework.

[achetal01]

Andrew I m not unfortunately. But Paulo has already conducted a threat model for that. I just added a couple more... LEt me find my comments and Ic an email those to you.

Thank you

On Tue, Oct 11, 2022 at 7:26 AM Andrew Martin @.***> wrote:

@pjbgf https://github.com/pjbgf @stefanprodan https://github.com/stefanprodan are either of you and/or any of the team free at 1300 BST tomorrow for the EMEA TAG Security meeting? We have space to run through an initial threat modelling session for this proposal with our new lightweight framework.

— Reply to this email directly, view it on GitHub https://github.com/cncf/tag-security/issues/896#issuecomment-1274781804, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARO764QXNNDLXWHERU6M7TLWCV2KPANCNFSM5UKX2W7A . You are receiving this because you were mentioned.Message ID: @.***>

[achetal01]

Here is my comment on the assessment

Reviewed the project makes sense and the security assessment has addressed key threats. Addressing Multi tenancy and cross platform continuous config validation with runtime is the next step in the evolution of that project... Only one comment is they must also provide API so that once an alert is generated the API can be used to auto respond to some of the config drifts in runtime, which could be extensive work, including re-instantiating the instance of app containers...automatically with correct configurations as aligned with Git. Also need to add detection around the Flagger if that gets compromised that could impact all the clusters and what Role/account the flagger has on each of the clusters, ideally non privileged Read Only access.... Hope this helps. Thanks Aradhna

On Tue, Oct 11, 2022 at 9:14 AM A C @.***> wrote:

Andrew I m not unfortunately. But Paulo has already conducted a threat model for that. I just added a couple more... LEt me find my comments and Ic an email those to you.

Thank you

On Tue, Oct 11, 2022 at 7:26 AM Andrew Martin @.***> wrote:

@pjbgf https://github.com/pjbgf @stefanprodan https://github.com/stefanprodan are either of you and/or any of the team free at 1300 BST tomorrow for the EMEA TAG Security meeting? We have space to run through an initial threat modelling session for this proposal with our new lightweight framework.

— Reply to this email directly, view it on GitHub https://github.com/cncf/tag-security/issues/896#issuecomment-1274781804, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARO764QXNNDLXWHERU6M7TLWCV2KPANCNFSM5UKX2W7A . You are receiving this because you were mentioned.Message ID: @.***>

[pjbgf]

@sublimino unfortunately that time conflicts with Flux's community meeting this week. When would be your next available slot?

[pjbgf]

@achetal01 I will take a look on the comments you left and come back to you.

[rowan-baker]

@sublimino unfortunately that time conflicts with Flux's community meeting this week. When would be your next available slot?

Responding on behalf of @sublimino, next available slot is December 7th 1PM GMT. Does that work for you?

[pjbgf]

@rowan-baker yes, that would be awesome, please book us in.

[rowan-baker]

Booked in for 7th December 1pm GMT.

[sublimino]

Thanks to everybody that contributed, the Flux threat model working document is here

[pjbgf]

@sublimino thank you (and the tag security) for all the help through the process. We will take a look and let you know whether we have any questions.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[makkes]

Quick status update: The Flux team is tracking the first recommendation from the threat model doc in two RFCs:

• https://github.com/fluxcd/flux2/pull/3366
• https://github.com/fluxcd/flux2/pull/3368

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[JustinCappos]

I'm picking this issue up and trying to understand why this hasn't moved to completion.

@pjbgf I just want to confirm that you're still interested in seeing this move along. Your assessment documents look pretty detailed so you've clearly put in a lot of work. There will be some back and forth to come, but we can try to push this forward and get the assessment completed. Please confirm your side still has the resource to work on this and wants to see it move forward.

[pjbgf]

@JustinCappos thank you for the heads up. @makkes and I will take a look on the outstanding work and get back to you in the coming weeks.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[JustinCappos]

polite nudge for @pjbgf

[pjbgf]

Hey @JustinCappos thank you for the nudge. @makkes and I caught up and went through the doc above once more. Our understanding was that the work left here was to review the recommendations. Here's our take on each one of the ones targeting the project:

1. Enforce HTTPS for Pod-to-Pod communication between Flux controllers: this work is being tracked as part of: https://github.com/fluxcd/flux2/pull/3368.
2. Investigate impact of multi-tenancy lockdown mode by default on multi-tenant setup: we believe the multi-tenancy implementation example mitigates this, as there isn't really an automated way to deploy Flux in "multi-tenancy mode".
3. Restrict permissions to reconciler Service Account (now cluster-admin) according to least privilege for soft multi-tenancy mode, i.e. granting impersonation to Flux-created SAs (K8s RBAC + OPA/Kyverno policy): this is already done as part of the multi-tenancy example, which also includes some Kyverno policies.
4. Use notification controller for user/admin awareness of Flux configuration/misconfiguration in teams/slack: the controller already notifies on some types of misconfiguration which could cause some reconciliations to fail. It supports a wide ranging of providers, which already includes Teams and Slack. The work on tracking notifications for security related misconfigurations we created the issue: https://github.com/fluxcd/notification-controller/issues/620.

Based on the above, we think we can close this issue as the work left is being tracked directly into the project's backlog.

CC: @hiddeco @makkes

[JustinCappos]

Okay, so is it fair to say that you have a self assessment and want to go through the joint assessment process at some point? We've a few items to take care of but can slot you in sometime in about a month, if interested.

[anvega]

Going ahead in closing this issue as the work left is being tracked directly into the project's backlog. Should the project team be interested in a deeper joint assessment after the issues have been addressed, please feel free to open up an issue requesting the assessment.