Where to host cs.k8s.io

[ameukam]

https://cs.k8s.io is running on a baremetal server provided by Equinix Metal(ex Packet) under CNCF budget and operated until now by @dims.

The question was asked about whether or not we should host CodeSearch on aaa cluster.

Ref: https://kubernetes.slack.com/archives/CCK68P2Q2/p1615204807111900?thread_ts=1615189697.108500&cid=CCK68P2Q2

Issue open to track the discussions and the consensus about this.

[nikhita]

@dims where is the original source code for cs.k8s.io? :eyes:

[nikhita]

/wg k8s-infra

[ameukam]

/sig contributor-experience /priority backlog

/assign @spiffxp cc @mrbobbytables @alisondy @cblecker @munnerz

[ameukam]

@dims where is the original source code for cs.k8s.io? eyes

@nikhita You can find the config here https://github.com/dims/k8s-code.appspot.com/

[BenTheElder]

What's the argument against hosting it on AAA?

[dims]

@BenTheElder nothing other than someone has to do it :) oh, i don't know how to wire the ingress/dns stuff

i tried a long time ago :) https://github.com/kubernetes/k8s.io/pull/96

[ameukam]

What's the argument against hosting it on AAA?

I would say lack of artifact destined for aaa (aka no up-to-date container image for hound). We could host the image on k8s-staging-infra-tools.

[nikhita]

@ameukam should this issue be migrated to the k/k8s.io repo?

[ameukam]

@nikhita I'm not sure about the right place of this issue. Just wanted to put this under SIG Contribex TLs and Chairs radar.

[BenTheElder]

it should be under k/k8s.io imho. I think we should host it on AAA fwiw.

[nikhita]

Moving to k8s.io repo. slack discussion - https://kubernetes.slack.com/archives/CCK68P2Q2/p1623300972130500

[spiffxp]

/sig contributor-experience /wg k8s-infra

[jimdaga]

I took a stab at onboarding codesearch; @spiffxp could I get your input? I want to make sure I didn't miss anything. I want to stage all the infra, and get it deployed via prow first. Then we can follow up with another PR to cut-over DNS when we are ready.

https://github.com/kubernetes/k8s.io/pull/2513 https://github.com/kubernetes/test-infra/pull/23201

I could also work on adding the docker build logic after, but I haven't worked in that repo yet so I'll have to do some digging.

cc @dims

[spiffxp]

/priority important-soon /milestone v1.23

[justaugustus]

What about using https://sourcegraph.com/kubernetes to minimize the maintenance burden here? This is something I suggested to @dims in the past, but didn't have the bandwidth to do at the time.

[dims]

choices are:

1. leave things where they are
2. move to k8s wg infra
3. redirect to cs.k8s.io to sourcegraph

• i have been taking care of 1 already for a while with minimal downtime, so i am ok with continuing to do so
• if someone wants to do 2, i am happy to work with help, show how things are setup and we can shut down the equinix vm
• i personally don't like option 3, i love the the hound UX, if the consensus is we should go with 3, that is fine with me. I am happy to run a personal instance on a custom domain for myself (community is welcome to use)

if i missed any other options, please feel free to chime in.

[spiffxp]

/unassign

[jimdaga]

FYI: If choice 2 is picked, my two PRs are pretty much ready to stage codesearch in the aaa cluster. There are a few small things that need to happen after the merge, but that's documented in my PRs.

• https://github.com/kubernetes/k8s.io/pull/2513
• https://github.com/kubernetes/test-infra/pull/23201

[dims]

thanks @jimdaga

+1 to give #2 a shot. will let Aaron and Arnaud to review and merge all 3 PRs

[ameukam]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[nikhita]

@ameukam what is remaining here?

[ameukam]

@ameukam what is remaining here?

Deploy a canary instance from https://github.com/kubernetes/k8s.io/pull/2513. Once we have confidence with that instance we can rollout a prod instance.

[Priyankasaggu11929]

/assign

@nikhita, I'm interested in helping with setting up a canary instance.

[Priyankasaggu11929]

Post-merge checklist item from PR https://github.com/kubernetes/k8s.io/pull/2513 that need working on:

• [ ] Publish cs-fetch-repo docker image (Open PR: https://github.com/kubernetes/test-infra/pull/25576)
• [ ] Update deployment to use deployed docker image (using a temp image for now)
• [X] After testing is completed, cutover DNS to new K8s hosted IP. (Done by https://github.com/kubernetes/k8s.io/pull/3416)

@pmgk07, once we're done with having https://github.com/kubernetes/test-infra/pull/25576 merged for adding cs-fetch-repos image under k8s infra, the next step would be updating the codesearch/deployment.yaml#L27 to use above hosted image.

[ameukam]

Update deployment to use deployed docker image (using a temp image for now)

@Priyankasaggu11929 Let's give @jimdaga the final call about this. There are possible changes that need to be added the Docker image.

[jimdaga]

Now that https://github.com/kubernetes/k8s.io/pull/3492 is merged, I see codesearch is deployed in the cluster!

However, it looks like the init containers are crashing:

kubectl get pods -n codesearch
NAME                         READY   STATUS                  RESTARTS   AGE
codesearch-5b975d449-lgm9b   0/1     Init:CrashLoopBackOff   8          19m
codesearch-5b975d449-zzqkl   0/1     Init:CrashLoopBackOff   8          19m

I'm out of the office right now, so I can't do a full debug. But it does seem like something needs fixing :( (I also don't have access to view pod logs, so not sure how to get that)

[Priyankasaggu11929]

Let's give @jimdaga the final call about this. There are possible changes that need to be added the Docker image.

+1. Yes 🙂

There's also an error for decoding ingress in the build-logs of the post-k8sio-deploy-app-codesearch job.

I've raised a minor patch fix: https://github.com/kubernetes/k8s.io/pull/3502

[ameukam]

Now that #3492 is merged, I see codesearch is deployed in the cluster!

However, it looks like the init containers are crashing:

kubectl get pods -n codesearch
NAME                         READY   STATUS                  RESTARTS   AGE
codesearch-5b975d449-lgm9b   0/1     Init:CrashLoopBackOff   8          19m
codesearch-5b975d449-zzqkl   0/1     Init:CrashLoopBackOff   8          19m

I'm out of the office right now, so I can't do a full debug. But it does seem like something needs fixing :( (I also don't have access to view pod logs, so not sure how to get that)

You can use GCP Logging console for the logs: https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.namespace_name%3D%22codesearch%22;cursorTimestamp=2022-03-11T06:20:53.646489047Z?project=kubernetes-public.

I did a quick research based on the logs and it suggested the issue may be related to the architecture of the Docker image.

 skopeo inspect docker://jdagostino2/codesearch-fetch:0.1.7 | jq .Architecture
"arm64"

The image seems to be built using a arm64 processor but the GKE nodes are amd64. We should try to switch to gcr.io/k8s-staging-infra-tools and see what's happening.