Add more S3 regions

[BenTheElder]

South America is the obvious missing one, but we should also consider adding others.

xref: https://github.com/kubernetes/k8s.io/pull/4739#discussion_r1101899000 and discussion in https://kubernetes.slack.com/archives/CCK68P2Q2/p1678504635523609

/sig k8s-infra /priority important-soon

When we add a bucket it will have to be added to the terraform config in k8s.io including the s3 to s3 replication, then once the bucket is reasonably well populated we'll want to add it to both https://github.com/kubernetes/registry.k8s.io/blob/8408d0501a88b3d2531ff54b14eeb0e3c900a4f3/cmd/archeio/app/buckets.go (AWS clients) and https://github.com/kubernetes/k8s.io/blob/5821271f875cc970b5701983e5695a025b5d952f/infra/gcp/terraform/k8s-infra-oci-proxy-prod/terraform.tfvars (non-GCP non-AWS clients)

This is much more relevant after #147

[ameukam]

/assign

/area infra/aws /milestone v1.27

[k8s-ci-robot]

@ameukam: The label(s) area/infra/aws cannot be applied, because the repository doesn't have them.

In response to this:

/assign

/area infra/aws /milestone v1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

We should ideally have:

1. All regions with notable client usage within AWS, for the AWS IP / region => bucket mapping
2. All regions that map closely to our Cloud Run / AR GCP regions, for external clients with the Cloud Run instance => bucket mapping

We started with a good approximation of 1) in https://github.com/kubernetes/registry.k8s.io/issues/39 / https://github.com/kubernetes/registry.k8s.io/issues/72, but it might be worth revisiting.

For 2) we have an okay approximation reusing the same set from 1), but we know there's at least the South America gap and should definitely re-evaluate and select the remaining regions that would most closely match now that we've moved to sending non-AWS-non-GCP traffic to S3 generally.

[ameukam]

List of remaining regions we can add:

• ap-east-1
• ap-northeast-2
• ap-southeast-1
• ap-southeast-3
• ap-southeast-4
• eu-central-1
• eu-central-2
• ca-central-1
• sa-east-1

[BenTheElder]

There's an inflection point where adding a region is wasteful.

If we are matching the IP to region with 1) for users within AWS and it only represents a very small amount of traffic within AWS then it's probably not worth it for 1), more regions than Cloud Run is not useful for 2) since we lean on the GCLB routing to determine approximate region for external traffic.

[BenTheElder]

OK so we looked at the costs today and storage costs are likely going to be smaller than traffic in basically any AWS region with actual traffic.

So just adding them for 1) is going to be generally worthwhile.

xref: https://kubernetes.slack.com/archives/CCK68P2Q2/p1678915249434949

[BenTheElder]

ref: https://github.com/kubernetes/registry.k8s.io/issues/181 for hold a bit on production changes

[dims]

cc @rothgar

[ameukam]

New AR regions added:

• asia-east2: https://github.com/kubernetes/k8s.io/pull/4949
• europe-west2: https://github.com/kubernetes/k8s.io/pull/4949
• southamerica-east1: https://github.com/kubernetes/k8s.io/pull/4924
• us-west3: https://github.com/kubernetes/k8s.io/pull/4419
• us-west4: https://github.com/kubernetes/k8s.io/pull/4419

[dims]

xref - some data from @TerryHowe here: https://kubernetes.slack.com/archives/CCK68P2Q2/p1681223002601649

[TerryHowe]

Adding more regions would be easy, but cloudfront should maybe be considered.

[BenTheElder]

There's another issue tracking CloudFront.

I think the idea is we still want to route in-AWS traffic with S3 in any region where cost to store < cost to serve cross region / cloud front. Which should be many regions.

CloudFront is probably the right answer for egress otherwise?

[BenTheElder]

Agree it should be easy, but we've avoided anything altering production behavior through securing the GCR redirect #181

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

Egress cost for the month of November 2023:

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/milestone v1.31