k8s.io icon indicating copy to clipboard operation
k8s.io copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

Redirect dl.k8s.io traffic to a community-owned GCS bucket instead of kubernetes-release

Open spiffxp opened this issue 4 years ago • 26 comments

Part of umbrella issues:

  • to deprecate and migrate away from gs://kubernetes-release: https://github.com/kubernetes/k8s.io/issues/2396
  • to migrate away from GCP project google-containers: https://github.com/kubernetes/k8s.io/issues/1571

Traffic gets to gs://kubernetes-release in one of two ways:

  • uri's that have the bucket name hardcoded (e.g. https://cs.k8s.io/?q=kubernetes-release%5B%5E-%5D&i=nope&files=&repos=)
  • dl.k8s.io uri's that redirect to the bucket (too many matches for cs.k8s.io to display, but e.g. used in kubernetes/kubernetes CHANGELOGs)

We discussed the idea of cutting over dl.k8s.io traffic first:

  • perhaps we could end up cutting over a large chunk of traffic sooner vs. later
  • this will help us better plan how to migrate traffic that has the bucket name hardcoded

The sketch is something like:

  • decide which project/bucket to use?
  • sync from old bucket to new bucket
  • change dl.k8s.io to use new bucket instead of old bucket
  • see what the delta in traffic is
  • decide how to keep buckets in sync?

Which project/bucket to use:

  • we have k8s-release hosting gs://k8s-release
  • we could choose to use a bucket hosted in k8s-artifacts-prod (simplifies billing)
  • we could choose to block until we have similar staging/promotion process for GCS artifacts

How to sync:

  • https://cloud.google.com/storage-transfer is pretty simple/fast to setup, I would at least use this for initial sync
    • There is currently no gcloud command, but there is an API and some bindings (ref: https://cloud.google.com/storage-transfer/docs/create-manage-transfer-program#cloud-to-cloud)
    • It supports recurring jobs, but no more frequent than hourly (ref: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob)
    • It supports one-time jobs, with a limit of 1K per day per project (ref: https://cloud.google.com/storage-transfer/quotas)
  • For ongoing sync we could
    • rely on hourly sync
    • periodically create one-time sync jobs more frequently
    • use gcs pubsub notifications to create sync jobs
    • alter release tooling to publish to multiple buckets
    • alter release tooling to create sync jobs

Seeing the delta in traffic:

  • we'll be able to see an influx of traffic in our billing report
  • @thockin and I need to decide if/how to share the existing traffic gs://kubernetes-release is getting

My preferences would be:

  • delete/recreate the gs://k8s-release bucket in k8s-artifacts-prod
  • use sts, rely on hourly sync job first, then prowjob that periodically creates one-time syncs

/assign @thockin since you were part of the discussion /assign @justaugustus @hasheddan For @kubernetes/release-engineering input

/wg k8s-infra /sig release /sig testing /area artifacts /milestone v1.21

spiffxp avatar Jan 22 '21 18:01 spiffxp

(Opened https://github.com/kubernetes/k8s.io/pull/1857 for dl.k8s.io/ci.)

justaugustus avatar Mar 31 '21 19:03 justaugustus

I manually initiated a one-time transfer from gs://kubernetes-release/release to gs://k8s-release/release following instructions at https://cloud.google.com/storage-transfer/docs/create-manage-transfer-console to get a sense of size/cost of transfer

Screen Shot 2021-07-09 at 2 00 56 PM

spiffxp avatar Jul 09 '21 21:07 spiffxp

delete/recreate the gs://k8s-release bucket in k8s-artifacts-prod

@justaugustus I would still like us to do this before we consider flipping over dl.k8s.io for real

I could be convinced to flip over temporarily to get an estimate of the amount of traffic this flips, but we should ultimately be using a similar approach as we do for GCR, with promotion from a staging bucket etc.

/assign @saschagrunert @puerco @cpanato @jeremyrickard other SIG Release leads for input

spiffxp avatar Jul 09 '21 21:07 spiffxp

/milestone v1.23 Definitely not doing this before v1.22 goes out the door.

I've pointed more things at dl.k8s.io as part of migrating away from hardcoding kubernetes-release-dev in URIs.

I'm going to edit the description to put this under the umbrella issue I've created for gs://kubernetes-release deprecation and migration

spiffxp avatar Jul 27 '21 04:07 spiffxp

We have concerns that we should hold off on this until we tackle cross-cloud promotion

Would also like to understand why the cost of this is so much larger than cost of container images

spiffxp avatar Sep 15 '21 20:09 spiffxp

Make a followup issue for dl.k8s.io log analysis, or look into temporarily enabling access logs and provide a sanitized version

spiffxp avatar Sep 15 '21 20:09 spiffxp

Blocked on https://github.com/kubernetes/k8s.io/issues/1375

spiffxp avatar Sep 29 '21 19:09 spiffxp

/milestone v1.24

spiffxp avatar Nov 24 '21 00:11 spiffxp

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Feb 22 '22 01:02 k8s-triage-robot

/remove-lifecycle stale

ameukam avatar Feb 22 '22 09:02 ameukam

/lifecycle frozen /milestone clear

ameukam avatar May 12 '22 02:05 ameukam

/remove-lifecycle frozen /milestone v1.26

ameukam avatar Aug 26 '22 15:08 ameukam

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Nov 24 '22 16:11 k8s-triage-robot

/remove-lifecycle stale /milestone v1.27 /remove-priority /priority important-longterm

Pending AWS credits (announced here) and decision from Fastly.

ameukam avatar Nov 24 '22 19:11 ameukam

@ameukam: Those labels are not set on the issue: priority/important-longterm

In response to this:

/remove-lifecycle stale /milestone v1.27 /remove-priority /priority important-longterm

Pending AWS credits (announced here) and decision from Fastly.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Nov 24 '22 19:11 k8s-ci-robot

Postponed to v1.28

/milestone v1.28

ameukam avatar Feb 22 '23 09:02 ameukam

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar May 23 '23 10:05 k8s-triage-robot

/remove-lifecycle stale

ameukam avatar May 25 '23 07:05 ameukam

We're still not always redirecting away from the bucket and we have most traffic going to the bucket directly (some of which is just skipping the redirect because users have come to depend on the GCS bucket).

We need to: a) ensure dl.k8s.io never redirects to the bucket b) switch publishing to a new bucket, restrict access to fastly only, and send users through dl.k8s.io => cdn.dl.k8s.io

BenTheElder avatar Jul 28 '23 16:07 BenTheElder

/milestone v1.29

ameukam avatar Sep 04 '23 14:09 ameukam

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Mar 07 '24 12:03 k8s-triage-robot

/remove-lifecycle stale

ameukam avatar Mar 07 '24 13:03 ameukam

/milestone v1.31

ameukam avatar Apr 18 '24 07:04 ameukam

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jul 17 '24 07:07 k8s-triage-robot

/remove-lifecycle stale

ameukam avatar Jul 17 '24 12:07 ameukam