atmos icon indicating copy to clipboard operation
atmos copied to clipboard

feat: Atmos List Deployments, Pro Deployments Upload

Open milldr opened this issue 6 months ago • 13 comments

what

New command for atmos list deployments

Includes option for --upload

  • ~~If --drift-enabled is true, only return deployments with settings.pro.drift_detection.enabled: true~~
  • If --upload, send the result to Atmos Pro

New arg for atmos terraform plan, --upload

  • Upload the result of a Terraform plan to Atmos Pro

Other changes:

  • Remove the logger that the pro client was using and replace with charmbracelet logger
  • Begin to refactor schema in to multiple files

why

atmos list deployments

  1. We want to build drift detection into Atmos Pro. In order to do this, we need to upload an object similar to the describe affected object. This object needs specific component deployments for specific stacks
  2. We do not have an existing type for the combination of a component and a stack. We previously have discussed introducing the term "Deployments". This PR creates that schema and term officially

~~Combine these concepts~~ for drift detection: atmos list deployments --upload

This will find all deployments, filter to drift enabled, then upload to Atmos Pro

atmos terraform plan --upload

  • When we trigger a plan with Atmos Pro for drift detection, we need to return the result of whether or not there is drift to AP

remove logger

We want to use the canonical logging that the rest of the application is using

schema refactor

The single schema.go file was getting extremely long (failing lint warnings) and difficult to navigate. First steps to start to refactor that out into a file per domain for the schema.

references

  • https://github.com/orgs/cloudposse-corp/discussions/275
  • DEV-3257

screenshots

CleanShot 2025-05-22 at 17 42 22@2x

Summary by CodeRabbit

  • New Features

    • Introduced a CLI command to list deployments, with optional upload of deployment data to a remote API.
    • Added support for uploading deployment status after running Terraform plan, including drift detection results.
    • Enhanced deployment listing to filter and display only deployments with specific features enabled.
    • Output formats for deployment listings now support both styled tables and CSV for non-TTY environments.
    • Added new configuration schemas for Atlantis integration, custom CLI commands, markdown styling, and vendored components.
  • Bug Fixes

    • Improved error handling and standardized error messages for deployment listing and status upload operations.
  • Tests

    • Added comprehensive unit and integration tests for deployment listing, upload functionality, CLI output validation, and Terraform execution with deployment status upload.
  • Chores

    • Updated and added configuration and manifest files for new test scenarios and improved test coverage.
    • Refactored and reorganized internal data structures for deployment, workflow, and configuration schemas.
    • Removed embedded logger dependencies from API client code and tests for simplification.
    • Enhanced Git repository interface abstraction for deployment status upload.

milldr avatar May 15 '25 20:05 milldr

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar May 15 '25 20:05 mergify[bot]

Codecov Report

:x: Patch coverage is 62.39316% with 264 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 56.98%. Comparing base (e494ebc) to head (da27903). :warning: Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
pkg/list/list_instances.go 53.36% 94 Missing and 3 partials :warning:
pkg/git/git.go 0.00% 35 Missing :warning:
internal/exec/pro.go 55.38% 26 Missing and 3 partials :warning:
pkg/pro/api_client.go 86.07% 17 Missing and 5 partials :warning:
cmd/list_instances.go 33.33% 18 Missing :warning:
pkg/pro/api_client_instance_status.go 60.00% 12 Missing and 6 partials :warning:
pkg/pro/api_client_instances.go 58.33% 10 Missing and 5 partials :warning:
pkg/list/format/table.go 0.00% 12 Missing :warning:
internal/exec/terraform.go 79.16% 9 Missing and 1 partial :warning:
pkg/utils/slice_utils.go 92.72% 3 Missing and 1 partial :warning:
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1254      +/-   ##
==========================================
+ Coverage   56.71%   56.98%   +0.26%     
==========================================
  Files         280      284       +4     
  Lines       29708    30286     +578     
==========================================
+ Hits        16850    17259     +409     
- Misses      11045    11200     +155     
- Partials     1813     1827      +14     
Flag Coverage Δ
unittests 56.98% <62.39%> (+0.26%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar May 15 '25 20:05 codecov[bot]

@milldr make sure look and feel reflects the other list commands. Also, I think @Cerebrovinny created an interface so we get the same behavior, look, and feel to atmos list commands. It was not fully adopted yet.

osterman avatar May 17 '25 23:05 osterman

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar May 19 '25 22:05 mergify[bot]

[!IMPORTANT]

Cloud Posse Engineering Team Review Required

This pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes.

To expedite this process, reach out to us on Slack in the #pr-reviews channel.

mergify[bot] avatar May 22 '25 22:05 mergify[bot]

[!WARNING]

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues. Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

mergify[bot] avatar May 22 '25 22:05 mergify[bot]

[!WARNING]

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues. Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

mergify[bot] avatar May 22 '25 23:05 mergify[bot]

[!WARNING]

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues. Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

mergify[bot] avatar May 27 '25 14:05 mergify[bot]

📝 Walkthrough

Walkthrough

Adds Atmos Pro integration: a new Pro API client interface with HTTP-backed upload methods (instances and instance status), GitRepo abstraction, "atmos list instances" CLI with optional Pro uploads, Terraform plan upload/status flows, schema reorganizations, centralized errors, and many tests/fixtures.

Changes

Cohort / File(s) Summary
Pro API client & DTOs
pkg/pro/api_client.go, pkg/pro/api_client_instances.go, pkg/pro/api_client_instance_status.go, pkg/pro/*_test.go, pkg/pro/dtos/*
Introduces AtmosProAPIClientInterface, adds HTTPClient field, implements UploadInstances and UploadInstanceStatus, centralizes error wrapping/logging, updates constructors to remove logger param, updates tests to pointer DTOs, adds TraceID and instance DTOs.
List instances feature & tests
cmd/list_instances.go, pkg/list/list_instances.go, pkg/list/*instances*.go, pkg/list/*_test.go, tests/snapshots/*, tests/test-cases/atmos-pro.yaml, tests/fixtures/scenarios/atmos-pro/*
New atmos list instances command and ExecuteListInstancesCmd: collect/sort instances, filter Pro drift-enabled ones, render TTY table or CSV, optionally upload instances to Pro; extensive unit/integration tests, fixtures and snapshots.
Terraform plan upload/status & exec flows
internal/exec/terraform.go, internal/exec/pro.go, internal/exec/pro_test.go, internal/exec/describe_affected.go, internal/exec/terraform_test.go
Adds --upload-status flag parsing, ensures --detailed-exitcode for plans, computes exit codes, introduces uploadStatus/shouldUploadStatus helpers that use Git info and ATMOS_PRO_RUN_ID to upload status to Pro; updates lock/unlock DTOs to pointers and wraps errors.
Git abstraction
pkg/git/git.go
Adds GitRepoInterface and DefaultGitRepo with GetLocalRepoInfo, GetRepoInfo, GetCurrentCommitSHA and contextual error wrapping.
Schema reorg & new types
pkg/schema/... (e.g., schema.go, instance.go, pro.go, atlantis.go, command.go, markdown.go, workflow.go, vendor_component.go, version.go)
Splits schema into focused files; adds/adjusts types including Instance, ProSettings, GithubOIDCSettings, Atlantis, Command, Markdown, Version, Workflow, VendorComponent, StackLockActionParams; adjusts mapstructure tags.
Errors & utils
errors/errors.go, pkg/utils/slice_utils.go, pkg/utils/slice_test.go, internal/exec/validate_utils.go, pkg/config/utils.go
Expands centralized error constants/messages; refactors slice utilities (rename, type-asserting variant, flag/value removal helpers); updates call sites/tests to use new errors and fmt-wrapped error formats.
List formatter tweak
pkg/list/format/table.go, pkg/list/list_values_test.go
Prioritizes nested vars map for key extraction and value lookup in table output; tests updated accordingly.
Config loading & constants
pkg/config/load.go, pkg/config/const.go
Adds ATMOS_PRO_RUN_ID and UploadStatusFlag constants; binds additional Pro env vars in loader.
Fixtures & CLI testcases
tests/fixtures/scenarios/atmos-pro/*, tests/test-cases/atmos-pro.yaml, tests/snapshots/*
Adds atmos-pro scenario fixtures (manifests, mixins, .gitignore), CLI test-case YAML for list instances, and golden snapshots.
Misc CLI wiring & tests
cmd/terraform_commands.go, cmd/cmd_utils.go, pkg/downloader/get_git_test.go, assorted tests
Wires upload-status flag into terraform commands, adjusts custom-command defaults, updates git-related error expectations, and adapts many tests to new signatures/error constants.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor U as User
  participant CLI as atmos
  participant TF as terraform
  participant G as GitRepo
  participant PRO as AtmosProAPIClient
  participant API as Pro API

  U->>CLI: atmos terraform plan --upload-status ...
  CLI->>TF: run plan (ensure --detailed-exitcode)
  TF-->>CLI: exit code
  alt upload-status set AND shouldUploadStatus AND exitCode in {0,2}
    CLI->>G: GetLocalRepoInfo / GetCurrentCommitSHA
    G-->>CLI: repo info + sha
    CLI->>PRO: NewAtmosProAPIClientFromEnv(...)
    CLI->>PRO: UploadInstanceStatus(dto)
    PRO->>API: PATCH /repos/{owner}/{repo}/instances/{stack}/{component}
    API-->>PRO: 200 OK
    PRO-->>CLI: success
    CLI-->>U: treat 0/2 as non-error
  else not eligible
    CLI-->>U: return original plan result
  end
sequenceDiagram
  autonumber
  actor U as User
  participant CLI as atmos
  participant L as list.instances
  participant DESC as DescribeStacks
  participant G as GitRepo
  participant PRO as AtmosProAPIClient
  participant API as Pro API

  U->>CLI: atmos list instances [--upload]
  CLI->>L: ExecuteListInstancesCmd(...)
  L->>DESC: DescribeStacks()
  DESC-->>L: stacks data
  L->>L: collect instances, filter pro-enabled
  alt TTY
    L-->>U: render table
  else non-TTY
    L-->>U: output CSV
  end
  opt --upload AND instances exist
    L->>G: GetLocalRepoInfo
    G-->>L: repo info
    L->>PRO: NewAtmosProAPIClientFromEnv(...)
    L->>PRO: UploadInstances({repo, instances})
    PRO->>API: POST /instances
    API-->>PRO: 200 OK
    PRO-->>L: success
  end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

  • cloudposse/atmos#1278 — modifies Atmos Pro API client, OIDC token exchange, and NewAtmosProAPIClientFromEnv signatures; strong overlap with client/constructor changes.
  • cloudposse/atmos#1358 — central errors/logging refactor; overlaps with new centralized errors and wrapping used throughout.
  • cloudposse/atmos#797 — earlier list command work; related to the new list instances feature and CLI wiring.

Suggested reviewers

  • osterman
  • milldr

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Out of Scope Changes Check ⚠️ Warning The PR contains substantial changes beyond implementing TF plan uploads, including large schema restructures (removals in pkg/schema/schema.go followed by many new schema files such as pkg/schema/atlantis.go, command.go, markdown.go, vendor_component.go, workflow.go, version.go), broad additions to the public errors surface, new public utilities and interfaces (GitRepoInterface, slice utils), and numerous unrelated tests and snapshots; these expand the public API and surface area in ways not required by DEV-3257. Because these disparate changes affect public types and package interfaces, they should be split out or justified separately to avoid coupling unrelated review scope with the plan-upload feature. Please separate non-essential refactors and large schema/API additions into one or more follow-up PRs or clearly document why each is necessary for the plan-upload work; if they must remain together, add an explicit rationale in the PR description and a compatibility note for public type changes, and run the full integration and API-compatibility tests before merge.
Docstring Coverage ⚠️ Warning Docstring coverage is 46.83% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title "feat: Atmos List Instances, Pro Upload" is concise and accurately highlights the primary work—adding a list-instances command and Atmos Pro upload functionality—so it reflects the main change in the changeset. It is short, specific, and readable for a teammate scanning history.
Linked Issues Check ✅ Passed DEV-3257's goal to implement Terraform plan upload is addressed: the PR adds the --upload-status flag and parsing logic, implements uploadStatus/shouldUploadStatus control paths, introduces Pro DTOs (InstancesUploadRequest, InstanceStatusUploadRequest), implements API client methods UploadInstanceStatus and UploadInstances, wires ATMOS_PRO_RUN_ID environment binding, and includes unit tests covering the upload flows. These changes align with the linked issue's primary objective to enable plan-status uploads to Atmos Pro.
✨ Finishing touches
  • [ ] 📝 Generate Docstrings
🧪 Generate unit tests
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment
  • [ ] Commit unit tests in branch feat/atmos-list-upload

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Jun 02 '25 19:06 coderabbitai[bot]

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar Jun 08 '25 16:06 mergify[bot]

/release

goruha avatar Jun 24 '25 21:06 goruha

/release

goruha avatar Jun 24 '25 22:06 goruha

/release

goruha avatar Jun 24 '25 22:06 goruha

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar Jul 02 '25 19:07 mergify[bot]

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar Jul 30 '25 16:07 mergify[bot]

💥 This pull request now has conflicts. Could you fix it @milldr? 🙏

mergify[bot] avatar Aug 06 '25 19:08 mergify[bot]

These changes were released in v1.188.0-test.0.

github-actions[bot] avatar Aug 19 '25 18:08 github-actions[bot]

These changes were released in v1.188.0-test.1.

github-actions[bot] avatar Aug 20 '25 18:08 github-actions[bot]

These changes were released in v1.188.0-test.2.

github-actions[bot] avatar Aug 20 '25 18:08 github-actions[bot]

These changes were released in v1.189.0-test.0.

github-actions[bot] avatar Sep 02 '25 15:09 github-actions[bot]

These changes were released in v1.190.0-test.0.

github-actions[bot] avatar Sep 05 '25 19:09 github-actions[bot]

These changes were released in v1.190.0-test.1.

github-actions[bot] avatar Sep 10 '25 15:09 github-actions[bot]

These changes were released in v1.190.0-test.2.

github-actions[bot] avatar Sep 10 '25 16:09 github-actions[bot]

These changes were released in v1.190.0-test.3.

github-actions[bot] avatar Sep 10 '25 16:09 github-actions[bot]

These changes were released in v1.190.0-test.4.

github-actions[bot] avatar Sep 10 '25 18:09 github-actions[bot]

These changes were released in v1.190.0-test.5.

github-actions[bot] avatar Sep 10 '25 18:09 github-actions[bot]

@coderabbitai resolve

milldr avatar Sep 11 '25 19:09 milldr

✅ Actions performed

Comments resolved and changes approved.

coderabbitai[bot] avatar Sep 11 '25 19:09 coderabbitai[bot]

@coderabbitai full review

milldr avatar Sep 11 '25 19:09 milldr

✅ Actions performed

Full review triggered.

coderabbitai[bot] avatar Sep 11 '25 19:09 coderabbitai[bot]