JenTing
JenTing
From the Jaeger tracing, the observation is that we call the `finalizeWorkspaceContent` almost within the same time. case 1. case 2.
We need to check the jaeger tracing on the gen63 cluster to see if it still happens or not.
> Message: Operation cannot be fulfilled on pods "prebuild-69cd1a21-2d04-4f27-8528-f605fc580986": the object has been modified; please apply your changes to the latest version and try again It's because multiple workers are...
I check the spans and corresponding logs, a case happens in prebuild, and the remote branch does not exist anymore.
> Still seeing this issue with prebuilds [[1](https://console.cloud.google.com/logs/query;cursorTimestamp=2022-08-10T00:23:00Z;query=severity%3DERROR%0AjsonPayload.serviceContext.service%3D%2528%22ws-manager%22%2529%0Aresource.labels.cluster_name%20:%20%2528%22us60%22%2529%0Atimestamp%3D%222022-08-10T00:23:00Z%22%0AinsertId%3D%223h5dii30dob9njbx%22;summaryFields=:false:32:beginning:false;timeRange=PT30S?project=workspace-clusters)][[2](https://console.cloud.google.com/logs/query;cursorTimestamp=2022-08-10T00:23:00Z;query=severity%3DERROR%0AjsonPayload.serviceContext.service%3D%2528%22ws-manager%22%2529%0Aresource.labels.cluster_name%20:%20%2528%22us60%22%2529%0Atimestamp%3D%222022-08-10T00:22:59Z%22%0AinsertId%3D%22wktpy6mbq5mmidex%22;summaryFields=:false:32:beginning:false;timeRange=PT30S?project=workspace-clusters)] in `us60` The way to reproduce it is when running prebuild, the remote branch is deleted at that moment.
I checked the `InitWorkspace` span, and I found an interesting thing, all the checkout branch error comes with the users using the [additionalRepositories](https://github.com/gitpod-io/website/blob/main/gitpod/docs/references/gitpod-yml.md#additionalrepositories) configuration within the _.gitpod.yml_. Also, I create...
From the pprof I dumped, the Goroutine leaks seem to happen in `http(*persistConn)readLoop` and `http(*persistConn)writeLoop`.
We still see the Goroutines increase in the us54 cluster. Reopen it.
The unit tests fail.
> > Hi, @filipjnc. Thanks for your report. This issue is already scheduled but has not yet been worked on due to priorities. Can I ask you to share the...