Enhanced OCI Registry Authentication and Layer Extraction Support

[cmharden]

What

This PR enhances Atmos's OCI vendoring capabilities by implementing authentication support for private registries and robust layer extraction for multiple OCI artifact formats. The changes resolve the 401 Unauthorized error when pulling from private registries and support both TAR and ZIP layer formats.

Why

Previously, Atmos's OCI vendoring was limited to:

• Only GitHub Container Registry (ghcr.io) authentication via GITHUB_TOKEN
• Single artifact type support (application/vnd.atmos.component.terraform.v1+tar+gzip)
• TAR-only layer extraction, causing failures with ZIP-based OCI artifacts

This resulted in:

• 401 Unauthorized errors when pulling from private registries
• Failed vendoring for OpenTofu modules and other ZIP-based artifacts
• Limited compatibility with enterprise OCI registries

References

• closes #369

Solution Overview

1. Multi-Source Authentication Framework

Implemented a comprehensive authentication system that attempts credentials from multiple sources in order of preference:

1. GitHub Container Registry - Uses GITHUB_TOKEN environment variable
2. Docker Credential Store - Integrates with ~/.docker/config.json and credential helpers
3. Environment Variables - Supports custom registry credentials via REGISTRY_NAME_USERNAME/PASSWORD
4. Cloud Provider Auth - Placeholder framework for AWS ECR, Azure ACR, Google GCR

2. Enhanced Layer Extraction

Added support for multiple OCI layer formats:

• TAR extraction - Original method for .tar and .tar.gz layers
• ZIP extraction - New support for archive/zip media types
• Raw data fallback - Robust fallback for unknown formats

3. Extended Artifact Type Support

Added support for additional OCI artifact types:

• application/vnd.atmos.component.terraform.v1+tar+gzip (original)
• application/vnd.opentofu.modulepkg (OpenTofu modules)
• application/vnd.terraform.module.v1+tar+gzip (Terraform modules)

Files Changed

Core Implementation

• internal/exec/oci_utils.go - Enhanced authentication and extraction logic

Test Coverage

• internal/exec/oci_utils_test.go - Comprehensive test suite (new file)

Key Changes

Authentication Enhancements

New Functions Added:

• getRegistryAuth() - Centralized authentication logic
• getDockerAuth() - Docker config file integration
• getCredentialStoreAuth() - Docker credential helper support
• decodeDockerAuth() - Base64 auth string decoding
• getECRAuth(), getACRAuth(), getGCRAuth() - Cloud provider placeholders

Enhanced pullImage() Function:

// Before: Only ghcr.io support
if strings.EqualFold(registry, "ghcr.io") {
    // GitHub token logic only
}

// After: Multi-source authentication
auth, err := getRegistryAuth(registry)
if err != nil {
    opts = append(opts, remote.WithAuth(authn.Anonymous))
} else {
    opts = append(opts, remote.WithAuth(auth))
}

Layer Extraction Improvements

New Functions Added:

• extractZipFile() - ZIP archive extraction
• extractRawData() - Raw data fallback extraction

Enhanced processLayer() Function:

// Before: TAR-only extraction
if err := extractTarball(uncompressed, destDir); err != nil {
    return fmt.Errorf("tarball extraction error: %w", err)
}

// After: Format-aware extraction
mediaTypeStr := string(mediaType)
if strings.Contains(mediaTypeStr, "zip") {
    extractionErr = extractZipFile(uncompressed, destDir)
} else {
    extractionErr = extractTarball(uncompressed, destDir)
}

Artifact Type Support

Enhanced checkArtifactType() Function:

// Before: Single artifact type
if manifest.ArtifactType != targetArtifactType {
    log.Warn("OCI image does not match the target artifact type")
}

// After: Multiple supported types
supportedTypes := []string{
    targetArtifactType,
    opentofuArtifactType,
    terraformArtifactType,
}

Test Coverage

Created comprehensive test suite covering:

Authentication Tests

• GitHub Container Registry (with/without token)
• Custom registry environment variables
• Docker config with direct auth
• Docker credential store integration
• Cloud provider authentication placeholders

Extraction Tests

• ZIP file extraction with multiple files/subdirectories
• Raw data extraction fallback
• Error handling for invalid formats

Integration Tests

• Full OCI image processing flow
• Error handling for invalid image references
• Performance benchmarks

Test Results

=== RUN   TestGetRegistryAuth (0.49s) - PASS
=== RUN   TestExtractZipFile (0.00s) - PASS  
=== RUN   TestExtractRawData (0.00s) - PASS
=== RUN   TestCheckArtifactType (0.00s) - PASS
=== RUN   TestDecodeDockerAuth (0.00s) - PASS
=== RUN   TestGetCredentialStoreAuth (0.00s) - PASS
=== RUN   TestCloudProviderAuth (0.00s) - PASS
=== RUN   TestProcessOciImageIntegration (1.40s) - PASS

Performance Benchmarks

BenchmarkExtractZipFile-14           148           8185153 ns/op
BenchmarkGetRegistryAuth-14     22547790                54.36 ns/op

Usage Examples

Environment Variable Authentication

# For registry: my-registry.com
export MY_REGISTRY_COM_USERNAME="username"
export MY_REGISTRY_COM_PASSWORD="password"

Docker Credential Store

# Uses existing Docker credentials from ~/.docker/config.json
docker login my-registry.com

GitHub Container Registry

# Uses existing GITHUB_TOKEN
export GITHUB_TOKEN="ghp_..."

Security Considerations

• No hardcoded credentials - All authentication is external
• Docker credential integration - Leverages existing secure credential storage
• Environment variable support - Follows security best practices
• Fallback to anonymous - Graceful degradation when no auth available

Testing Instructions

Manual Testing

1. Private Registry Test:

   # Set up credentials
   export MY_REGISTRY_COM_USERNAME="testuser"
   export MY_REGISTRY_COM_PASSWORD="testpass"
   
   # Test vendoring
   atmos vendor pull
   

2. Docker Credential Test:

   # Login to registry
   docker login my-registry.com
   
   # Test vendoring (should use Docker credentials)
   atmos vendor pull
   

3. OpenTofu Module Test:

   # Vendor an OpenTofu module (ZIP format)
   atmos vendor pull oci://registry.defenseunicorns.com/navy-pbme/terraform-aws-uds-s3irsa:v1.0.0
   

Automated Testing

# Run OCI-specific tests
go test ./internal/exec -v -run "TestGetRegistryAuth|TestExtractZipFile|TestExtractRawData|TestCheckArtifactType"

# Run performance benchmarks
go test ./internal/exec -bench "BenchmarkExtractZipFile|BenchmarkGetRegistryAuth" -run "^$"

Future Enhancements

The authentication framework is designed for easy extension:

1. AWS ECR Integration - Ready for AWS SDK integration
2. Azure ACR Integration - Ready for Azure SDK integration
3. Google GCR Integration - Ready for Google Cloud SDK integration
4. Additional Artifact Types - Easy to add new OCI artifact formats

Breaking Changes

None - This is a purely additive enhancement that maintains backward compatibility.

Checklist

• [x] Enhanced OCI authentication for private registries
• [x] Added ZIP layer extraction support
• [x] Extended artifact type compatibility
• [x] Comprehensive test coverage
• [x] Performance benchmarks
• [x] Documentation and usage examples
• [x] Security considerations addressed
• [x] Backward compatibility maintained
• [x] Error handling and logging improved

Result

This PR transforms Atmos's OCI vendoring from a basic GitHub-only feature into a robust, enterprise-ready solution that supports:

• Any private OCI registry with flexible authentication
• Multiple artifact formats (Atmos, OpenTofu, Terraform)
• Multiple layer formats (TAR, ZIP, raw data)
• Comprehensive error handling and fallback mechanisms
• Full test coverage ensuring reliability

Summary by CodeRabbit

• New Features

  • Multi-source OCI registry authentication (GitHub, Docker config/creds, env vars, AWS ECR, Azure ACR, Google GCR) with anonymous fallback; new OCI configuration settings (GitHub/Azure/Docker) exposed.

• Improvements

  • More resilient OCI image processing with per-layer reporting and tolerance for partial failures.
  • Hardened ZIP extraction with path-traversal and size protections.
  • Expanded artifact-type recognition (Terraform/OpenTofu).

• Tests

  • Extensive tests for auth paths, extraction, parsing, and processing.

• Chores

  • Dependency upgrades and added environment-variable bindings for OCI credentials.

[coderabbitai[bot]]

📝 Walkthrough

Walkthrough

Adds multi-source OCI registry authentication (GHCR, Docker, env, ECR, ACR, GCR), threads OCI settings into image pull flow, implements per-layer extraction with ZIP/TAR dispatch and hardened ZIP protections, extends schema/config for OCI settings, bumps AWS SDK deps, and adds extensive auth/extraction tests.

Changes

Cohort / File(s)	Summary
Core orchestration & extraction internal/exec/oci_utils.go	Integrates atmosConfig into pull flow; adds getRegistryAuth, bindEnv, per-layer processing, media-type dispatch (ZIP/TAR), structured logging, handleExtractionError, non-fatal partial-layer handling, and ZIP helpers (extractRawData, ZIP guards).
Provider auth resolvers internal/exec/oci_github.go, internal/exec/oci_google.go, internal/exec/oci_aws.go, internal/exec/oci_azure.go, internal/exec/oci_docker.go	New provider auth helpers: GHCR (env/atmos), GCR (ADC), ECR (AWS SDK token), ACR (service-principal & default credential token-exchange), and Docker (config.json + credential helpers). Adds public DockerConfig type.
Extraction helpers & tests internal/exec/...*, internal/exec/oci_extraction_test.go	Adds ZIP extraction flow (extractZipFile, entry validation, path traversal guards, maxZipSize) and raw-data extraction; unit tests for valid/malicious ZIPs and raw-data extraction.
Auth & processing tests internal/exec/oci_auth_test.go, internal/exec/oci_github_test.go, internal/exec/oci_google_test.go, internal/exec/oci_aws_test.go, internal/exec/oci_azure_test.go, internal/exec/oci_docker_test.go, internal/exec/oci_processing_test.go	Adds table-driven/unit tests for registry auth selection/preference, cred decoding, registry detection, ECR/ACR parsing/token flows, and processOciImage/pullImage behaviors.
Config & schema pkg/config/load.go, pkg/schema/schema.go	Adds OCI env bindings in setEnv and a new exported OCI OCISettings field (Github/Azure tokens and DockerConfig) in AtmosSettings.
Deps go.mod	Bumps aws-sdk-go-v2 modules and adds service/ecr dependency.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Caller
  participant OCI as pullImage/processOciImage
  participant Auth as getRegistryAuth
  participant Viper as Viper/Env
  participant Docker as Docker config/cred helpers
  participant ECR as AWS ECR
  participant ACR as Azure ACR
  participant GCR as Google GCR

  Caller->>OCI: pullImage(imageRef, atmosConfig)
  OCI->>Auth: getRegistryAuth(registry, atmosConfig)
  Auth->>Viper: check GHCR/env/atmos vars
  Auth->>Docker: check docker config.json & cred helpers
  Auth->>ECR: attempt ECR auth (AWS SDK)
  Auth->>ACR: attempt ACR auth (service-principal/default)
  Auth->>GCR: attempt GCR (ADC)
  Auth-->>OCI: authenticator or error
  alt authenticator returned
    OCI->>Registry: pull with auth
  else
    OCI->>Registry: pull anonymously
  end

sequenceDiagram
  autonumber
  participant OCI as processOciImage
  participant Layer as image layer
  participant ZIP as extractZipFile
  participant TAR as tar extractor
  participant RAW as extractRawData

  OCI->>Layer: iterate layers
  alt mediaType contains "zip"
    OCI->>ZIP: extractZipFile(reader, dest)
    alt success
      ZIP-->>OCI: files extracted
    else
      ZIP-->>OCI: error
      OCI->>RAW: extractRawData(reader, dest, idx)
    end
  else
    OCI->>TAR: extract tar
    alt success
      TAR-->>OCI: files extracted
    else
      TAR-->>OCI: error
      OCI->>RAW: extractRawData(reader, dest, idx)
    end
  end
  OCI-->>OCI: log per-layer outcomes and summary

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• cloudposse/atmos#714 — Modifies internal/exec/oci_utils.go and related OCI processing; likely overlaps on extraction and logging changes.

Suggested labels

minor

Suggested reviewers

• osterman
• aknysh
• kevcube

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues Check	⚠️ Warning	The PR delivers a robust multi-source authentication framework but omits the Terraform-specific credential mechanisms specified in issue #369, namely parsing the Terraform credentials file and supporting TF_TOKEN_<HOST> environment variables, so it does not fully satisfy the linked issue’s requirements.	Implement support for reading ~/.terraform.d/credentials.tfrc.json and environment variables of the form TF_TOKEN_<HOST> to fully comply with the Terraform private registry authentication requirements in issue #369.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly captures the two main enhancements—adding registry authentication and improving layer extraction—and directly reflects the primary changes in the PR without extraneous detail.
Out of Scope Changes Check	✅ Passed	All introduced code and tests pertain to the enhanced OCI authentication flow and layer extraction features described in the linked issue, with no evident additions that lie outside the defined objectives for private registry support.

✨ Finishing touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between e42b65999456c887888126e00f11128299842b74 and 682361222e10ca70f599985d7e166dd328c27ec5.

📒 Files selected for processing (2)

• pkg/config/load.go (1 hunks)
• pkg/schema/schema.go (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/config/load.go

🧰 Additional context used

📓 Path-based instructions (3)

pkg/**/*.go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

Place business logic in pkg rather than in cmd

Group related functionality in pkg/ subpackages by domain

Files:

• pkg/schema/schema.go

**/*.go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

**/*.go: All code must pass golangci-lint checks Follow Go error handling idioms and use meaningful error messages Wrap errors with context using fmt.Errorf("context: %w", err) Consider custom error types for domain-specific errors Follow standard Go coding style; run gofmt and goimports Use snake_case for environment variables Document complex logic with inline comments

**/*.go: All comments must end with periods - comments should be complete sentences (enforced by golangci-lint's godot linter) All errors MUST be wrapped using static errors defined in errors/errors.go; never use dynamic errors directly Always wrap errors with the static error first and add context-specific details after the static error; use %w verb to preserve error chain ALWAYS use viper.BindEnv() for environment variable binding, and EVERY env var MUST have an ATMOS_ alternative Distinguish between logging and UI output: only use logging for system/debug info, and use stderr/UI for prompts, progress, errors, and data for piping should go to stdout Most text UI MUST go to stderr; only data/results go to stdout for proper piping; use utils.PrintfMessageToTUI() for UI messages Atmos MUST work on Linux, macOS, and Windows—write portable implementations and prefer SDKs over calling binaries when available ALWAYS compile after making changes; verify no compilation errors; run tests after successful compile; fix compilation errors immediately before proceeding Write a test to reproduce the bug before fixing, run the test and confirm failure, fix the bug iteratively, and verify fix doesn't break existing tests

Files:

• pkg/schema/schema.go

**/!(*_test).go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

Document all exported functions, types, and methods with Go doc comments

Files:

• pkg/schema/schema.go

🧬 Code graph analysis (1)

pkg/schema/schema.go (1)

internal/exec/oci_docker.go (1)

• DockerConfig (30-36)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Summary

🔇 Additional comments (2)

pkg/schema/schema.go (2)

267-269: Field wiring checks out.

Solid addition—the OCI settings slot naturally into the existing config surface.

---

279-287: Struct definition looks good.

Clean schema with clear tags and docs; ready to support the new auth flows.

[!TIP]

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

• Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
• Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[osterman]

• [ ] We'll also need you to refactor the tests for each implementation so they are aligned with each provider

[osterman]

@cmharden have you seen:

• #1269

[cmharden]

@cmharden have you seen:

• Support vendoring for insecure (HTTP) OCI registries #1269

I haven't, but it sounds like a great idea. I'd be happy to pick it up once we get this PR across the line.

[osterman]

Please make sure to run golanglint-ci locally on the changes and gofumpt for formatting. Make sure all tests are passing, at least in our CI.

[cmharden]

Please make sure to run golanglint-ci locally on the changes and gofumpt for formatting. Make sure all tests are passing, at least in our CI.

Pushed

[mergify[bot]]

💥 This pull request now has conflicts. Could you fix it @cmharden? 🙏

[cmharden]

@osterman 👋 I think this is ready for review. Maybe...

[mergify[bot]]

💥 This pull request now has conflicts. Could you fix it @cmharden? 🙏

[cmharden]

@osterman bump...