toc
toc copied to clipboard
cncf
Argo Project Graduation Proposal
Given that the Security audit and the CII badges are marked as not completed, this project is not currently eligible for Graduation. The wording for CII is "Have achieved and maintained a Core Infrastructure Initiative badge", so it is clearly intended that they be completed before, not at the moment of graduation. Can we close this and re-apply after the criteria are met?
@justincormack - where are you looking for the badges please?
Argo Workflow has its badge: https://github.com/argoproj/argo-workflows Argo Events also has its badge: https://github.com/argoproj/argo-events
I'm not sure about CD/rollouts.
The text of the proposal says "Core Infrastructure Initiative Best Practices Badges have been completed for Argo Workflows and Events and are in progress for Argo CD and Rollouts."
Due to renames, its pretty confusing as the badge project does not seem to list workflows ona search for argo, and shows another (failing) entry as https://bestpractices.coreinfrastructure.org/en/projects/1446 so it needs cleaning up.
https://bestpractices.coreinfrastructure.org/en/projects?q=argo
@justincormack Thanks for your comments and attention. Would you be interested in co-sponsoring this proposal?
All four of the projects have passing badges now:
| Project | Badge |
|---|---|
| CD | |
| Rollouts | |
| Workflows | |
| Events |
Adding SIG App Delivery for review, can easily be changed if there's a better fit for a different SIG.
@amye @resouer i am happy to pick up from @michelleN to help with the process side of graduation here. thanks @michelleN !
@dims More than welcome! The current stage is DD doc is under drafting and we are trying to schedule interview meetings with end users from Argo, let's follow up in the slack channel.
Thank you, @dims! It will be great to have you. Will add you to the argo-graduation slack channel. And thank you @michelleN for all your help thus far.
update from TOC + DD:
https://lists.cncf.io/g/cncf-toc/message/5823
Hello everyone,
Argo project is applying for graduation status: PR: https://github.com/cncf/toc/pull/604 DD: https://docs.google.com/document/d/1R4WjMG9s9JX8onZvOzEFSjBBFAInurN8tSiAFLqj-FE/edit#heading=h.kd4eg2uz3lt0
DD has been reviewed by myself and SIG App Delivery. We've also conducted interviews with end users. We are supportive of Argo going into graduation. We are now calling for the 2 week public comment period prior to the vote.
I think the question I'm most interested in is understanding the positioning with respect to Flux. What is the intended messaging to CNCF End Users in terms of which solution to pick?
My understanding is that CNCF does not pick winners/losers, but instead lets multiple flowers bloom and lets the users decide. If so, CNCF does not need to make any specific recommendation.
3rd party security review docs. Also linked from DD doc and this proposal text. https://github.com/argoproj/argoproj/blob/master/docs/argo_threat_model.pdf https://github.com/argoproj/argoproj/blob/master/docs/argo_security_final_report.pdf @justincormack
+1 on this!
https://github.com/thoth-station/ is heavily using Argo Workflows. We have two usage patterns: 1. very short life (kind of) batch processing, depending on the season its >20k workflows a week, and 2. long-running (which is still <20min) workflows based on user/bot requests, generating responses to API calls.
We would love to see the project go fwd, and our experience over the past close to two years is pretty good: responsiveness and openness of the community is good!
+1 for Argo, we have been using Argo Workflows with Open Data Hub (https://github.com/opendatahub-io) and Kubeflow Pipelines (https://github.com/kubeflow) for a while. The community has always been very helpful and would love to see this project graduate.
[Writing on behalf of the TOC]. We are seeing lots of usage of Argo by end users which is a very good sign, but remain concerned about the security posture of the project. We would like to see increased focus on security before graduation, including the assessment that has started, and engagement with the Security Buddy program with TAG Security.
In the interest of not leaving the PR open indefinitely, we recommend closing this one and re-opening when the security posture is clearer and issues from the previous audit have been addressed. (CNCF could instigate a follow-up audit to clarify that the security position has been improved.)
Thanks for the update @lizrice. Can the TOC please give some more guidance on your security concerns? We have addressed the long and short term issues and recommendations from the external audit and have engaged with our security pal as well as kicking off a review with the Security TAG.
From the Argo community vantage point, we have some concerns over the process changes and the resultant delays. When we reached out at the start of the graduation process in February 2021, we were told to not do a STAG review and the guidance was that the external audit is more comprehensive. After we completed the external audit and addressed the issues identified, the STAG review and security pal were added as requirements in July, adding 3-4 mo of waiting in the review queue. Up until now, there has not been a mention of a second external review, which will be another 3-4 month wait to get started, given the backlog of the external auditor. This is a huge surprise and the community feels like the goal posts keep getting moved!
The project is fully committed to security and we always have, and will, work diligently on any recommendations from external and internal audits, but as these audits take a very long time to schedule, a clear understanding of this process would greatly help with scheduling and planning.
With the ask to close the PR and abandon the graduation process, we’d also like to understand what that would entail moving forward. e.g. do we need to start from the very beginning, and find new sponsors, complete the external security audit, redo the user interview etc. or is there an abridged process given all that has already been done so far? It would greatly help if we can get clear guidance on the graduation process, next steps, requirements, owners and expected timelines before we restart the process.
As I understand it the Trail of Bits review recommended a further assessment after the identified issues had been addressed, so that's not a new suggestion. The TOC has a broader concern, that (as indicated by the audit) security needs to be more closely considered as part of the "culture" of the project. It's not just a question of fixing the issues that have been identified, it's also about making sure that the project carefully considers the security implications going forward. This is especially crucial for a project like Argo that's so intertwined with the software supply chain. The recommendation to work with TAG Security and get a Security Buddy is intended to help address this.
The recommendation to close this PR doesn't mean that you have to throw away the work so far and start again, although the sponsor might want to do some "refresh" e.g. speak to some more end users. It's really so that we are all clear that the TOC isn't ready to pass a graduation vote at this time. I don't see any reason why this same PR couldn't be re-opened to indicate when you think the security culture of the project is more mature and deserves another look from the TOC.
TAG-Security is working with Argo now on a security joint review - that is being kicked off now. This will help educate us in how the security pals effort can be directed to benefit the Argo project. Review leads: @jlk @IAXES.
Ref: https://github.com/cncf/tag-security/issues/739
@lizrice after reviewing with the rest of the project I think there's a lot happening on security that is simply not as visible as it should be so we'd like to help make sure all of that is visible and I think it will go a long way to show how security is embedded into the culture of the project.
Before we close the PR, I think we can update on that.
Folks, it's been a few months, how far did you all get with the last round of feedback? thanks!
Security has always been important to the Argo project and the 100s of companies that use our projects in production, as they are very often critical components of platforms and infrastructure. Spurred by the initial comments in the graduation PR almost a year ago and the TOC comments through LIz above, we have been increasing our efforts not only to strengthen our security, but also to make sure the community and our users are aware of the efforts. Below is a summary of work that the project has completed or is in the process of completing.
- We fixed all relevant security issues that were brought up in the external security assessment last year, many before the report was even published.
- A SIG Security group has been formed within the project to align between the four projects and raise and prioritize any security related issues or concerns
- The project security policies and contacts have been reviewed to make sure escalation paths are clear and that there are clear responsibilities and associated maintainers
- Together with CNCF, we have completed a project to integrate fuzzing through OSS-Fuzz and there are now 33 fuzzers that are now running against our code, with very good results. Big thanks to CNCF for sponsoring this!
- We are working through the RFP process with CNCF to ensure that the issues and recommendations from the previous assessment have been addressed, but also to get on to a path of continuity as our projects move fast and add features at a rapid pace.
- an additional external security review to be completed ASAP
- a yearly external security review
- There is a security assessment underway with the CNCF Security STAG, as a complement to the external assessments, to ensure alignment with any CNCF best practices and recommendations
- The project was assigned a security buddy from the security TAG, which has met with the team and is involved in the Security TAG assessment.
- And lastly, the work that has been completed has shown that our security is good, both in terms of absolute numbers and in comparison to other CNCF projects, but it has not been well-communicated. To address this we have a series of blogs planned that will go into much more detail on the status of all our efforts. We hope to have the first one out before the end of the month.
A lot of work has gone into, and will continue to go into, making and keeping all the Argo projects secure and we are thankful for the resources that have been made available to us from the CNCF!
Security has always been important to the Argo project and the 100s of companies that use our projects in production, as they are very often critical components of platforms and infrastructure. Spurred by the initial comments in the graduation PR almost a year ago and the TOC comments through LIz above, we have been increasing our efforts not only to strengthen our security, but also to make sure the community and our users are aware of the efforts. Below is a summary of work that the project has completed or is in the process of completing.
Thanks!
- We fixed all relevant security issues that were brought up in the external security assessment last year, many before the report was even published.
Got it.
- A SIG Security group has been formed within the project to align between the four projects and raise and prioritize any security related issues or concerns
Is this already documented somewhere? that is easy to find? How are on it? (from which companies?)
- The project security policies and contacts have been reviewed to make sure escalation paths are clear and that there are clear responsibilities and associated maintainers
Same as above, can you please share urls?
- Together with CNCF, we have completed a project to integrate fuzzing through OSS-Fuzz and there are now 33 fuzzers that are now running against our code, with very good results. Big thanks to CNCF for sponsoring this!
Nice!
We are working through the RFP process with CNCF to ensure that the issues and recommendations from the previous assessment have been addressed, but also to get on to a path of continuity as our projects move fast and add features at a rapid pace.
- an additional external security review to be completed ASAP
- a yearly external security review
What state of the RFP process are we in? Has it gotten to the point of selecting vendors?
- There is a security assessment underway with the CNCF Security STAG, as a complement to the external assessments, to ensure alignment with any CNCF best practices and recommendations
This one right? https://github.com/cncf/tag-security/issues/554 Our newly elected TOC member @TheFoxAtWork indicated that it may take a few months for the 4 sub-projects in Argo
- The project was assigned a security buddy from the security TAG, which has met with the team and is involved in the Security TAG assessment.
Glad to hear this!
- And lastly, the work that has been completed has shown that our security is good, both in terms of absolute numbers and in comparison to other CNCF projects, but it has not been well-communicated. To address this we have a series of blogs planned that will go into much more detail on the status of all our efforts. We hope to have the first one out before the end of the month.
Is there a schedule for the blogs?
A lot of work has gone into, and will continue to go into, making and keeping all the Argo projects secure and we are thankful for the resources that have been made available to us from the CNCF!
+1
Some other notes:
- Argo cd : is https://argo-cd.readthedocs.io/en/stable/security_considerations/ slated to be replaced? (i see a deprecated notice) is there a plan to remove this page with better content?
- Argo rollouts : seems to indicate that 3 community members are to be emailed directly? https://argoproj.github.io/argo-rollouts/security/
- Argo Events : does not seem to have a security link in the main page in documentation
- Argo Workflows : same as Events
- Can https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/security.md be surfaced better in Argo CD documentation?
- Can https://github.com/argoproj/argo-workflows/blob/master/docs/security.md be surfaced better in Argo workflow documentation? (says email as mentioned above, but looks like Jesse's email is different)
- https://github.com/argoproj/argo-events/blob/master/SECURITY.md has emails listed as well (note that the website does not have this info as mentioned earlier)
- https://github.com/argoproj/argo-rollouts is missing a SECURITY.md but there is a https://github.com/argoproj/argo-rollouts/security/policy which says email 3 people
- Can there be a uniform process for all the projects? one single mailing list perhaps? (example containerd - https://github.com/containerd/project/blob/main/SECURITY.md)
- would also recommend outlining an embargo process and an announce list (see containerd above)
(edited out answered parts for brevity)
Is this already documented somewhere? that is easy to find? How are on it? (from which companies?)
This was discussed, decided and documented in our weekly maintainer meeting. Right now, it consists of volunteers from the maintainer group and have had representation from Intuit, Red Hat, Codefresh and Akuity, but the individuals havent been externalized other than in the meeting notes.
- The project security policies and contacts have been reviewed to make sure escalation paths are clear and that there are clear responsibilities and associated maintainers
Same as above, can you please share urls?
The security.md files never got updated. The PRs have been filed. Thanks for catching!
We are working through the RFP process with CNCF to ensure that the issues and recommendations from the previous assessment have been addressed, but also to get on to a path of continuity as our projects move fast and add features at a rapid pace.
- an additional external security review to be completed ASAP
- a yearly external security review
What state of the RFP process are we in? Has it gotten to the point of selecting vendors?
The process was started in Nov last year and we are still waiting for OSTIF to complete the RFP write-up, so we are not in selection yet.
- There is a security assessment underway with the CNCF Security STAG, as a complement to the external assessments, to ensure alignment with any CNCF best practices and recommendations
This one right? cncf/tag-security#554 Our newly elected TOC member @TheFoxAtWork indicated that it may take a few months for the 4 sub-projects in Argo
Yes. We have done the first few rounds of Q&A and information sharing with the STAG, so the actual reviews are planned to start in February, with each project taking 1-2w to complete.
- And lastly, the work that has been completed has shown that our security is good, both in terms of absolute numbers and in comparison to other CNCF projects, but it has not been well-communicated. To address this we have a series of blogs planned that will go into much more detail on the status of all our efforts. We hope to have the first one out before the end of the month.
Is there a schedule for the blogs?
The first one should be within a few weeks, as mentioned, and then the plan is to have them on a ~2-3w cadence.
Some other notes:
* Argo cd : is https://argo-cd.readthedocs.io/en/stable/security_considerations/ slated to be replaced? (i see a deprecated notice) is there a plan to remove this page with better content?
As you noted, the page has been deprecated in favor of the two github pages that hold the advisories and policy. The plan is to keep that content in github, but we'll look into improving that page as part of our upcoming doc re-write.
* Argo rollouts : seems to indicate that 3 community members are to be emailed directly? https://argoproj.github.io/argo-rollouts/security/ * https://github.com/argoproj/argo-events/blob/master/SECURITY.md has emails listed as well (note that the website does not have this info as mentioned earlier) * https://github.com/argoproj/argo-rollouts is missing a SECURITY.md but there is a https://github.com/argoproj/argo-rollouts/security/policy which says email 3 people * Can there be a uniform process for all the projects? one single mailing list perhaps? (example containerd - https://github.com/containerd/project/blob/main/SECURITY.md)
See above, this should have been fixed with updated security.md files. Should be corrected once the PRs are merged
* Argo Events : does not seem to have a security link in the main page in documentation * Argo Workflows : same as Events * Can https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/security.md be surfaced better in Argo CD documentation? * Can https://github.com/argoproj/argo-workflows/blob/master/docs/security.md be surfaced better in Argo workflow documentation? (says email as mentioned above, but looks like Jesse's email is different)
Duly noted. We'll look into updating the web pages with this as part of our planned refresh/re-write of the docs.
* would also recommend outlining an embargo process and an announce list (see containerd above)
Good idea. We'll bring that up in our next SIG Security meeting.