[POC] Manual integration release and signing

[dkirov-dd]

Summary

Implements a proof-of-concept for manually uploading integration wheels to S3 with TUF (The Update Framework) signing support, enabling secure distribution with cryptographic verification.

Key Features

• Enhanced Build Command: Generates pointer files with complete metadata (name, version, URI, digest, length, custom fields)
• Smart Upload Command:
  • Organizes files in S3 with proper structure (simple/, pointers/, metadata/)
  • Generates PEP 503 compliant indexes using constant-complexity approach (no wheel downloads)
  • Idempotent uploads skip re-uploading identical versions
• New Sign Command: Generates and signs TUF metadata (root, targets, snapshot, timestamp) with Ed25519 keys
• AWS-Vault Integration: Automatically handles authentication without verbose wrapper commands
• Integration Test Script: End-to-end workflow verification

Architecture

S3 Bucket Structure:

test-public-integration-wheels/
├── simple/
│   ├── index.html (root index)
│   └── datadog-{integration}/
│       ├── index.html (package index)
│       └── *.whl
├── pointers/
│   └── datadog-{integration}/
│       └── *.pointer
└── metadata/
    ├── root.json
    ├── targets.json
    ├── snapshot.json
    └── timestamp.json

Pointer File Format (YAML):

pointer:
  name: datadog-postgres
  version: 1.2.3
  uri: https://test-public-integration-wheels.s3.eu-north-1.amazonaws.com/simple/datadog-postgres/datadog_postgres-1.2.3-py3-none-any.whl
  digest: abc123...  # SHA256
  length: 12345      # Bytes
  custom: {}         # Reserved for in-toto

Usage

Before (verbose):

aws-vault exec sso-agent-integrations-dev-account-admin -- ddev release upload --public aws_neuron

After (simple):

ddev release build postgres
ddev release upload --public postgres  # Auto aws-vault!
ddev release sign --generate-keys      # Auto aws-vault!

Integration Test:

./scripts/test_tuf_workflow.sh postgres

Files Added

• datadog_checks_dev/datadog_checks/dev/tooling/simple_index.py (215 lines): PEP 503 index generation
• datadog_checks_dev/datadog_checks/dev/tooling/tuf_signing.py (393 lines): TUF metadata generation/signing
• datadog_checks_dev/datadog_checks/dev/tooling/commands/release/sign.py (119 lines): Sign command implementation
• datadog_checks_dev/datadog_checks/dev/tooling/aws_helpers.py (113 lines): AWS-vault integration helpers
• scripts/test_tuf_workflow.sh: End-to-end integration test script

Files Modified

• datadog_checks_dev/datadog_checks/dev/tooling/release.py: Enhanced pointer generation and upload logic
• datadog_checks_dev/datadog_checks/dev/tooling/commands/release/upload.py: Added aws-vault integration
• ddev/src/ddev/cli/release/__init__.py: Registered sign command

POC Limitations

• Dummy keys only: Uses auto-generated Ed25519 keys (not production-ready)
• No in-toto integration: Supply chain verification placeholder only
• Manual signing: ddev release sign must be run manually after uploads
• checks_downloader not modified: Out of scope for this POC

Future Work

• Integrate with checks_downloader to consume pointer files
• Implement production key management (HSM/KMS)
• Add in-toto supply chain verification
• Automate signing in CI/CD pipeline
• Add CDN for metadata distribution

Test Plan

• [x] Build integration creates pointer file with all required fields
• [x] Upload organizes files in proper S3 structure
• [x] Upload generates PEP 503 compliant indexes without downloading wheels
• [x] Sign command generates valid TUF metadata
• [x] TUF metadata properly signed with Ed25519 keys
• [x] AWS-vault integration works seamlessly
• [x] Integration test script validates end-to-end workflow

🤖 Generated with Claude Code

[github-actions[bot]]

⚠️ Recommendation: Add qa/skip-qa label

This PR does not modify any files shipped with the agent.

To help streamline the release process, please consider adding the qa/skip-qa label if these changes do not require QA testing.

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 91.15%. Comparing base (5cf5dc7) to head (2b3e4f5). :warning: Report is 48 commits behind head on master.

Additional details and impacted files

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.