Documenting request and response from TAGs for sandbox projects

[dims]

Discussion on slack https://cloud-native.slack.com/archives/C0MP69YF4/p1658858779093989

We should write this down somewhere.

[amye]

Bringing everything into one place: @dims: "Documenting request and response from TAGs for sandbox projects :thread: (spun off from the sandbox call today)" "Quite often during the sandbox review calls, we request TAGs to talk to the incoming projects and respond back to TOC in some form or fashion. The request(s) go out in a summary email from Amye and projects often watch the video of the sandbox call and figure out what is requested and how they respond. Seems like the TOC would like a written update(?) from either the TAG or the project or both(?) How can we streamline this so all parties involved know what is being asked from them? @toc-members Please chime in with your ideas." @cathyhongzhang: "Since Amye already captured what TOC needs from the project, we should just expect that the project provides a written update on the missing/additional info. It may not be a good thing to add a TAG in between to relay the information. The update needed for different projects is different. If we want to streamline this, we may need to classify the update into several categories." @amye: "They have the capacity to update their applications!" @dims: "Amye, if we update the sheet we can’t compare (unless we put it in another row?)" @amye: "So what could happen here: I move over the applications into accepted + not accepted after a sandbox review meeting, the project can reapply with the TAG feedback. Konveyor is a great example." "This gives us a written record." @TheFoxAtWork: "I like that process, lets codify it." @amye: "Also it’s much easier to tell what’s a reapply and what’s not." @dims: "cool, so when we start reviewing we look for these and bump them up ahead of the others. SGTM!"

[jberkus]

Why not use GH issues in this repo for the initial project questions? Speaking based on helping out the Konveyor team, "edit your application" is not a good workflow. It doesn't allow you to directly answer questions, it doesn't permit discussion or follow-up questions, and the appropriate TAG can't weigh in. And the applications are pretty hard for the TOC to read to see exactly what's been changed -- let alone what might have been changed in the project that there's no application question for.

My suggestion is: when the TOC has questions about an application, they can open an issue in this repo. The project will then answer those questions, possibly with the approprite TAG, and then the project will re-submit when all questions are answered. That's a workflow that would have saved Konveyor 6 months of waiting.

[dims]

For now the guidance/process is:

• Immediate after sandbox review, @amye will send one line summaries with accept/deny/more-info along with a link to the TOC playlist on youtube
• projects are welcome to review the recording to see what items TOC was debating about (when the recording is available on the playlist above later in the day)
• sandbox projects can use regular public or private communication methods to reach out to TOC folks if they need clarification on things
• when they are ready sandbox projects re-submit a fresh proposal to the TOC (same what they did earlier - filling up the google form) with a note that this is a resubmission.
• During the next sandbox review, TOC will prioritize the resubmissions and deal with them first.

@amye sounds good?

[jberkus]

... that sounds like the existing process, which has already been shown to be vulnerable to communications failures. What's the reason to never clearly communicate to the project the questions they need to answer? What is gained by making it difficult for the project to figure that out?

[joshgav]

Immediate after sandbox review, @amye will send one line summaries with accept/deny/more-info along with a link to the TOC playlist on youtube projects are welcome to review the recording to see what items TOC was debating about (when the recording is available on the playlist above later in the day)

Particularly for projects requiring further support/review, how about @amye opens an issue in the correct TAG's repo with these details and a timestamped link to the TOC discussion. We can choose a standard label like "TOC review" so these can be easily queried and aggregated across the github.com/cncf org.

TOC members and sponsors should still participate through issues in TAG repos. In fact this might help introduce TOC members to TAGs for better cooperation and awareness.

Long term maybe projects should be assigned to a TAG via an issue before they present to TOC so they can get some advice or insights in advance and save everyone time. Results and next steps could be tracked in the same issue.

[jberkus]

... or, heck, just assign a TOC member at the sandbox review meeting to send an email to the form submitter asking the questions, if dealing with this by GH issue isn't achievable right now.

"Listen to the meeting video to figure out what your project needs to clarify" just doesn't work. Those discussions are often unclear, and do not indicate any TOC consensus on what needs to be answered.

[linsun]

Hey folks!

I recently tried to take our bumblebee project through the sandbox process, and I appreciate all the reviewers to take the time to review the project. After going through the rejection, here are a few things I would I would share about our experience and some areas that could be improved:

• The wait time is quite long. I submitted the project in March and it didn't get reviewed till late July.
• No easy way to update the submission. We have also been invited by TAG runtime to present our project but we could not add that info as there is no obvious way to modify an existing submission. Yes, I could do another submission but I worry it would reset the the position in the queue. Would not it be easy if we use github issue (or PR) to submit the project for sandbox? That way, we can keep update it as needed.
• Is it possible to assign a TOC member or a TAG lead to work with the project owner briefly before the TOC review? This way, the project can be represented better.
• Reapply process isn't clear. Would be really helpful to have this clarified in the meeting mins that summarized the result of the sandbox review. Ideally, it would be nice to offer a TOC member to assist the reapply if the project really believe it should be part of CNCF sandbox projects.

Thanks for listening!

[TheFoxAtWork]

In response the community’s feedback, suggestions, and experiences in the sandbox process, the TOC is exploring the following changes. Over the next several weeks we will be establishing a new process, form, and tracking model to provide better granularity in feedback, increased transparency of the queue, and clear expectations for stakeholders of the sandbox process.

• We will be leveraging a dedicated sandbox repository
• The new repository will leverage a GitHub Issue Form to capture the existing content of the Google Form with additional clarity of questions and other factors to solicit improved awareness around the state of the project
• Clearly explain timeliness, expectations of the process, and other considerations
• Issues will be affixed with labels to relay their status (i.e. Approved, declined, need-info, assign-TAG, etc)
• Leverage a project board to transparently show the community what is in the backlog, where projects are at or queued, etc.
• Provide a GitHub query that shows in which order we will process the applications (and which ones we will prioritize before we take up things in that query)
• Provide instructions to TAGs for when a sandbox project is sent to them clean up any confusion or inconsistencies with existing sandbox instructional material

https://lists.cncf.io/g/cncf-toc/message/7351

[linsun]

Thanks so much for listening to feedback and improve the process, love it!!

[jberkus]

+1 NB

[TheFoxAtWork]

On today's TOC call, we invited the community to also propose changes, removals, and additions to the questions we asking applying projects, please feel free to add them here.

[jberkus]

I've created a new issue to specifically discuss the questions: https://github.com/cncf/toc/issues/901

[cathyhongzhang]

I like the idea of "Leverage a project board to transparently show the community what is in the backlog, where projects are at or queued, etc.". Currently, we have this link to show the sandbox projects in the backlog https://docs.google.com/spreadsheets/d/1Nnh_usr0tSZxaUpxTusqeIqKxMmvuEViRkyO9e_Do40/edit#gid=1398510353. Whether we will use this link as the project board or refactor this one into a new one, we'd better add the project board link to the sandbox application document so that the applicant can have a good idea about the time line.

[amye]

I like the idea of "Leverage a project board to transparently show the community what is in the backlog, where projects are at or queued, etc.". Currently, we have this link to show the sandbox projects in the backlog https://docs.google.com/spreadsheets/d/1Nnh_usr0tSZxaUpxTusqeIqKxMmvuEViRkyO9e_Do40/edit#gid=1398510353. Whether we will use this link as the project board or refactor this one into a new one, we'd better add the project board link to the sandbox application document so that the applicant can have a good idea about the time line.

So here, it's attached to the sandbox repo - we have a workflow set for any newly created card will be added to 'New'. https://github.com/orgs/cncf/projects/14

[LukasGentele]

@amye Are the projects in this google sheet still being worked on? We submitted DevSpace quite a while ago and it got pushed back during the last TOC meeting due to lack of time. Now, with the new issue-based process (love it btw!), I'm not sure if DevSpace will still be looked at during the next sandbox application review process or not. Can you confirm this?

[amye]

@amye Are the projects in this google sheet still being worked on? We submitted DevSpace quite a while ago and it got pushed back during the last TOC meeting due to lack of time. Now, with the new issue-based process (love it btw!), I'm not sure if DevSpace will still be looked at during the next sandbox application review process or not. Can you confirm this?

We'll be reviewing the google sheet at the next sandbox review meeting on December 13th.

[LukasGentele]

Thanks for confirming, @amye. That's great news. I appreciate the quick reply. Thanks!

[dims]

we are rocking and rolling here - https://github.com/cncf/sandbox/issues. When TOC requests a TAG to look at a sandbox project, we would log the back and forth if any in the issues in that repo.

closing this issue. thanks!