registry.k8s.io

registry.k8s.io copied to clipboard

Continuing from https://github.com/kubernetes/k8s.io/issues/1343#issuecomment-1165393785

Cloud Run logs requests btw. It looks like this:

{
  "insertId": "62c2d53e0003371fdc7d0e63",
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "https://oci-proxy-4txm7cz3ca-ew.a.run.app/v2/pause/manifests/latest",
    "requestSize": "359",
    "status": 308,
    "responseSize": "823",
    "userAgent": "curl/7.79.1",
    "remoteIp": "62.3.X.X",
    "serverIp": "216.239.36.53",
    "latency": "0.002888185s",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "revision_name": "oci-proxy-00007-rus",
      "service_name": "oci-proxy",
      "location": "europe-west1",
      "configuration_name": "oci-proxy",
      "project_id": "coen-mahamed-ali"
    }
  },
  "timestamp": "2022-07-04T11:55:42.210719Z",
  "severity": "INFO",
  "labels": {
    "instanceId": "00c527f6d4c262c0c5e52e02b9e0598455af84a75be4db07302d51c05ad943142fb40a1eedd66cc7d8fb9fed14ac8349f4a06236b5f8235e83c02c64ac3b494fa3"
  },
  "logName": "projects/coen-mahamed-ali/logs/run.googleapis.com%2Frequests",
  "trace": "projects/coen-mahamed-ali/traces/3c44b9996e13e00f68afa0b8dad552aa",
  "receiveTimestamp": "2022-07-04T11:55:42.217327936Z",
  "spanId": "6672162191736732715",
  "traceSampled": true
}

We can sink these logs to bigquery and start crunching data.

@BobyMCbobs @hh

/sig k8s-infra /priority important-soon

/retitle Access Logging registery.k8s.io

@upodroid @Riaankl, how should we coordinate the log processing with BQ?

Currently in ii's sandbox project, we have a dataset that we use for providing data to the community via a DataStudio report. It's generated through https://github.com/kubernetes/k8s.io/tree/main/images/public-log-asn-matcher. Perhaps the data could be combined in there? @Riaankl

I think we'd need some kinda link to be set up for allowing cross-account datasets, if it is possible.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

FWIW: I've been getting by OK so far with the cloud log querying.

If we want to produce some reports we can sink to BQ, but that itself costs money and it's unclear how useful those reports will be for driving future changes. We're primarily constrained by who actually provides us resources.

For visibility: This is the dashboard I'm using to track QPS: https://console.cloud.google.com/monitoring/dashboards/builder/07365a2f-9d9b-41af-860e-a3fa5e4164ad;tzId=America%2FLos_Angeles;startTime=2022-12-01T20:06:27.146Z;endTime=2023-02-09T20:06:27.147Z?mods=logs_tg_staging&project=k8s-infra-oci-proxy-prod&dashboardBuilderState=%257B%2522editModeEnabled%2522:false%257D

Which is the main metric I'm personally interested in seeing uptick in, especially after #143 (other than ... the billing report)

At the moment I don't think top images are a concern, they were useful for the k8s.gcr.io data for approaching the redirect but mostly a curiosity otherwise.

Now that it's relatively simple to add / remove regions and we know bandwidth costs far exceed storage costs in AWS at least, we'll likely be expanding regions and just watching the bills for the moment.

We do however have a bigquery sink in place for the production registry which we're not using yet, we need to be careful how we use this as there's lots of PII.

https://github.com/kubernetes/registry.k8s.io#privacy

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

We can close this out. Logs are being sinked to bq for future analysis.