Cloud Run Docker Mirror

Cloud Run Docker Mirror copies images from one Docker v2 Registry to another, as a service. With this service, you can:

• Mirror popular public images (e.g. postgres, redis) into your internal artifact registry.

• Reduce rate limiting or costs against public Docker registries by pulling images infrequently and caching them locally.

• Increase availability by reducing dependencies on third-party systems.

• Improve build times by bring images onto a local or higher-bandwidth network.

• Ensure consistency and enforce security practices/scanning without "getting in the way" of your developers.

Note that Cloud Run Docker Mirror is most applicable for long-running mirroring where the upstream repository is expected to live indefinitely. For one-time mirroring, consider using gcrane or other migration tools directly, without the overhead of a continuously running service.

The application is written in Go and is optimized for Cloud Run. It may work on other systems, but they are untested.

This is not an official Google product.

Quick start

Suppose we want to mirror postgres on the Docker Hub to our internal Artifact Registry repository:

1. Deploy the service to Cloud Run:

   gcloud run deploy "cloud-run-docker-mirror" \
     --quiet \
     --image "us-docker.pkg.dev/vargolabs/cloud-run-docker-mirror/server" \
     --platform "managed" \
     --project "${PROJECT_ID}" \
     --region "us-central1" \
     --timeout "30m"
   

2. Invoke the service with a JSON payload:

   URL=$(gcloud run services describe "cloud-run-docker-mirror" --project "${PROJECT_ID}" --platform "managed" --region "us-central1" --format 'value(status.url)')
   
   curl "${URL}" -d '{"mirrors": [{"src":"postgres", "dst":"us-docker.pkg.dev/my-project/my-repo/postgres"}]}'
   

3. Invoke this URL on a fixed schedule to mirror, perhaps using Cloud Scheduler.

Production setup

This section details a full production setup using Cloud Run, Cloud Scheduler, and Artifact Registry.

1. Install or update the Cloud SDK for your operating system. Alternatively, you can run these commands from Cloud Shell, which has the SDK and other popular tools pre-installed.

2. Authenticate to the SDK:

   gcloud auth login --update-adc
   

   This will open a browser and prompt you to authenticate with your Google account.

3. Create a Google Cloud project, or use an existing one.

4. Export your project ID as an environment variable. The rest of these instructions assumes this environment variable is set.

   export PROJECT_ID="my-project"
   

   Note this is your project ID, not the project number or name.

5. Enable the necessary Google APIs on the project - this only needs to be done once per project:

   gcloud services enable --project "${PROJECT_ID}" \
     appengine.googleapis.com \
     artifactregistry.googleapis.com \
     cloudscheduler.googleapis.com \
     run.googleapis.com
   

   This operation can take a few minutes, especially for recently-created projects.

6. Deploy the service to Cloud Run:

   gcloud run deploy "cloud-run-docker-mirror" \
     --quiet \
     --image "us-docker.pkg.dev/vargolabs/cloud-run-docker-mirror/server" \
     --platform "managed" \
     --project "${PROJECT_ID}" \
     --region "us-central1" \
     --timeout "15m"
   

7. Create a Service Account with permissions to invoke the Cloud Run service (this is required later):

   gcloud iam service-accounts create "docker-mirror-invoker" \
     --project "${PROJECT_ID}"
   

   gcloud run services add-iam-policy-binding "cloud-run-docker-mirror" \
     --quiet \
     --project "${PROJECT_ID}" \
     --platform "managed" \
     --region "us-central1" \
     --member "serviceAccount:docker-mirror-invoker@${PROJECT_ID}.iam.gserviceaccount.com" \
     --role "roles/run.invoker"
   

8. Enable AppEngine (required for Cloud Scheduler):

   gcloud app create \
     --quiet \
     --project "${PROJECT_ID}" \
     --region "us-central"
   

9. Get the URL of the deployed service:

   URL="$(gcloud run services describe "cloud-run-docker-mirror" \
     --quiet \
     --project "${PROJECT_ID}" \
     --platform "managed" \
     --region "us-central1" \
     --format 'value(status.url)')"
   

10. Build the JSON body:

    BODY="$(cat <<EOF
    {
      "mirrors": [
        {
          "src": "postgres",
          "dst": "us-docker.pkg.dev/${PROJECT_ID}/mirrors/postgres"
        },
        {
          "src": "redis",
          "dst": "us-docker.pkg.dev/${PROJECT_ID}/mirrors/redis"
        }
      ]
    }
    EOF
    )"
    

11. Create a Cloud Scheduler HTTP job to invoke the service daily at 7:00am:

    gcloud scheduler jobs create http "cloud-run-docker-mirror" \
      --project "${PROJECT_ID}" \
      --description "Mirror Docker repositories" \
      --uri "${URL}/" \
      --message-body "${BODY}" \
      --oidc-service-account-email "docker-mirror-invoker@${PROJECT_ID}.iam.gserviceaccount.com" \
      --schedule "0 7 * * *" \
      --time-zone="US/Eastern"
    

12. (Optional) Run the scheduled job now:

    gcloud scheduler jobs run "cloud-run-docker-mirror" \
      --project "${PROJECT_ID}"
    

Now use us-docker.pkg.dev/${PROJECT_ID}/mirrors/postgres instead of postgres in your CI, deployments, etc!

Request format

The request is a JSON request of the format:

{
  "mirrors": [
    {
      "src": "...",
      "dst": "...",
    }
  ]
}

Where:

• mirrors is an array of mirror requests

  • src is the source repository

  • dst is the destination repository

Response format

The response is a JSON body of the format:

{
  "ok": true | false,
  "errors": [
    {
      "index": 1,
      "name": "x to y",
      "error": "failed for some reason"
    }
  ]
}

Where:

• ok indicates overall success or failure

• errors is an array of errors

  • index is the index in the input request where the failure occurred

  • name is the name of the mirror

  • error is the actual error

Building the image

The Cloud Run Docker Mirror image is hosted and is available at multiple source including Artifact Registry and GitHub Packages:

• us-docker.pkg.dev/vargolabs/cloud-run-docker-mirror/server
• europe-docker.pkg.dev/vargolabs/cloud-run-docker-mirror/server
• asia-docker.pkg.dev/vargolabs/cloud-run-docker-mirror/server
• gcr.io/vargolabs/cloud-run-docker-mirror/server
• docker.pkg.github.com/sethvargo/cloud-run-docker-mirror/server

If you would like to build and publish your own package:

1. Clone this repository and change into that directory:

   git clone https://github.com/sethvargo/cloud-run-docker-mirror
   

   cd cloud-run-docker-mirror
   

2. Build the container:

   docker build -t your-org/your-repo/your-image:tag .
   

3. Push the container to your registry:

   docker push your-org/your-repo/your-image:tag