atmos icon indicating copy to clipboard operation
atmos copied to clipboard

feat: implement !append YAML function for list concatenation

Open osterman opened this issue 5 months ago β€’ 10 comments

what

  • Implements the !append YAML function that allows fine-grained control over list merging behavior in Atmos stack configurations
  • Lists tagged with !append will be concatenated with base values instead of replaced
  • Adds comprehensive unit tests and integration test fixtures

why

  • Resolves the ongoing challenge of needing to concatenate lists on a case-by-case basis
  • Currently, users have to fall back to using maps instead of lists when they need append behavior
  • This is particularly important for fields like depends_on where appending is often the desired behavior rather than replacement
  • The !append tag provides opt-in, per-field control that works alongside the global list_merge_strategy setting

Key Features

  • Opt-in behavior: Only lists explicitly tagged with !append use append mode
  • Works alongside global settings: The !append tag works independently of the global list_merge_strategy setting
  • Nested support: Works with deeply nested configurations
  • Backward compatible: No impact on existing configurations without the tag

Example Usage

# base.yaml
components:
  terraform:
    eks:
      settings:
        depends_on:
          - vpc
          - iam-role

# override.yaml
components:
  terraform:
    eks:
      settings:
        depends_on: !append  # This tag indicates append mode
          - rds
          - elasticache
          
# Result: depends_on = [vpc, iam-role, rds, elasticache]

Testing

  • βœ… All unit tests pass
  • βœ… Build succeeds without errors
  • βœ… Linting passes with no issues
  • βœ… Code follows Atmos conventions and patterns

references

πŸ€– Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added support for a !append YAML function to append items to lists during configuration merging (per-field, preserves order, works with nested structures and global list-merge strategies).
  • Tests
    • Added unit tests for append-tag utilities and merge behavior, plus integration-style test cases and expected results demonstrating !append semantics.
  • Documentation
    • Added docs and examples explaining !append usage, behavior, and interactions with existing merge settings.

osterman avatar Sep 25 '25 05:09 osterman

[!WARNING]

Rate limit exceeded

@osterman has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 5 minutes and 23 seconds before requesting another review.

βŒ› How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between ff9bca723c7fd7039db569c233ada54eaf6b32f5 and a0d392e1a2bc9de97477a361d25f8f40ea28b6d5.

β›” Files ignored due to path filters (2)
  • website/package-lock.json is excluded by !**/package-lock.json
  • website/pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
πŸ“’ Files selected for processing (1)
  • website/package.json (1 hunks)
πŸ“ Walkthrough

Walkthrough

Adds per-field YAML !append handling: YAML loader records tagged sequences as append-wrapped values; merge pre-processing consumes those wrappers to concatenate lists into merged maps. New helpers, tests, docs, test fixtures, a CLI test helper, and one new exported error were added. No public API signatures were changed.

Changes

Cohort / File(s) Summary
YAML processing
pkg/config/process_yaml.go
Decode YAML sequence nodes tagged !append, decode each list item, wrap with append metadata and store in Viper at the current path, then clear the node tag to avoid re-processing.
Merge logic
pkg/merge/merge.go
Add pre-merge pass processAppendTags that walks input maps, detects append-wrapped lists and nested maps, and merges/concatenates lists into the running merged map before the normal merge flow. New internal helpers: processAppendTags, processValue, processAppendList, processNestedMap. No change to public Merge signature.
Merge tests
pkg/merge/merge_append_test.go
New unit tests for per-field !append semantics and interaction with a global list-merge strategy across multiple input documents.
Append utilities
pkg/utils/yaml_func_append.go, pkg/utils/yaml_utils.go
New helpers and constant: ProcessAppendTag, IsAppendTag, HasAppendTag, ExtractAppendListValue, WrapWithAppendTag; AtmosYamlFuncAppend = "!append".
Utility tests
pkg/utils/yaml_func_append_test.go
Unit tests for tag detection, presence checks, extraction, wrapping, and ProcessAppendTag validation.
E2E test cases
tests/test-cases/append-function/base.yaml, tests/test-cases/append-function/override-with-append.yaml, tests/test-cases/append-function/expected-result.yaml
Added base config, override using !append, and expected merged result demonstrating appended lists.
Docs
website/docs/functions/yaml/append.mdx, website/docs/functions/yaml/index.mdx
New documentation and examples describing !append usage, semantics, and interaction with global list merge strategies.
Errors
errors/errors.go
Added exported error ErrRefusingToDeleteSymlink = errors.New("refusing to delete symbolic link").
Test harness
tests/cli_test.go
Added validateAtmosBinary(repoRoot string) (string, string) and integrated it into TestMain to surface a skip reason or the binary path for CLI-related tests.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant YAML as YAML Loader
  participant PY as process_yaml.go
  participant U as utils (append helpers)
  participant V as Viper (config store)

  YAML->>PY: Parse node
  alt node is sequence with tag "!append"
    PY->>PY: Decode each list item
    PY->>U: WrapWithAppendTag(items)
    U-->>PY: {"__atmos_append__": items}
    PY->>V: Set(path, wrappedValue)
    PY->>PY: Clear node.Tag
  else other node
    PY->>V: Set(path, decoded value)
  end
sequenceDiagram
  autonumber
  participant M as MergeWithOptions
  participant P as processAppendTags
  participant U as utils (append helpers)
  participant S as MergedState

  M->>P: Pre-process input map
  loop each key,value
    P->>P: processValue(key, value)
    alt value is append-wrapped list
      P->>U: ExtractAppendListValue(value)
      U-->>P: items
      P->>S: Append items into S[key] (create if missing)
    else value is nested map
      P->>P: processAppendTags(nestedMap)
      P->>S: Merge/assign nested map into S[key]
    else
      P->>S: Assign/override S[key] = value
    end
  end
  M->>M: Continue normal merge flow with processed map

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • cloudposse/atmos#1506 β€” modifies YAML custom-tag handling and clears node tags after processing; strongly related to the !append tag implementation.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Out of Scope Changes Check ⚠️ Warning The PR also adds an unrelated file-operation error constant (ErrRefusingToDeleteSymlink) and CLI binary validation logic in tests/cli_test.go, which are not part of the !append feature scope defined in DEV-2980. Please remove or relocate the new symlink error constant and CLI test helper changes into a separate pull request so that this PR focuses solely on the !append YAML functionality.
βœ… Passed checks (4 passed)
Check name Status Explanation
Description Check βœ… Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check βœ… Passed The title clearly and concisely describes the primary purpose of the changeset, which is adding the !append YAML function for list concatenation, matching the main feature implemented in the pull request.
Linked Issues Check βœ… Passed The pull request fully implements the per-field !append tag mechanism described in DEV-2980 by detecting and wrapping append-tagged lists in the YAML parser, merging them correctly in the merge logic, providing utility helpers and unit tests, and documenting usage, all of which satisfy the linked issue’s requirements.
Docstring Coverage βœ… Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❀️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Sep 25 '25 05:09 coderabbitai[bot]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 72.84%. Comparing base (8e75aa1) to head (3e8e709).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1513      +/-   ##
==========================================
+ Coverage   72.81%   72.84%   +0.02%     
==========================================
  Files         530      531       +1     
  Lines       50667    50715      +48     
==========================================
+ Hits        36892    36941      +49     
+ Misses      11002    11000       -2     
- Partials     2773     2774       +1     
Flag Coverage Ξ”
unittests 72.84% <100.00%> (+0.02%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Ξ”
errors/errors.go 100.00% <ΓΈ> (ΓΈ)
pkg/merge/merge.go 86.71% <100.00%> (+1.49%) :arrow_up:
pkg/utils/yaml_func_append.go 100.00% <100.00%> (ΓΈ)
pkg/utils/yaml_utils.go 81.10% <ΓΈ> (ΓΈ)

... and 2 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Sep 25 '25 05:09 codecov[bot]

πŸ’₯ This pull request now has conflicts. Could you fix it @osterman? πŸ™

mergify[bot] avatar Sep 26 '25 13:09 mergify[bot]

[!WARNING]

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues. Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

mergify[bot] avatar Dec 08 '25 21:12 mergify[bot]

πŸ’₯ This pull request now has conflicts. Could you fix it @osterman? πŸ™

mergify[bot] avatar Dec 08 '25 21:12 mergify[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-08-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 08 '25 21:12 github-actions[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-09-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 08 '25 21:12 github-actions[bot]

πŸ’₯ This pull request now has conflicts. Could you fix it @osterman? πŸ™

mergify[bot] avatar Dec 09 '25 04:12 mergify[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-09-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 09 '25 04:12 github-actions[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-17-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 09 '25 18:12 github-actions[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-18-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 17 '25 15:12 github-actions[bot]

πŸ’₯ This pull request now has conflicts. Could you fix it @osterman? πŸ™

mergify[bot] avatar Dec 18 '25 18:12 mergify[bot]

[!WARNING]

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-22-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

github-actions[bot] avatar Dec 18 '25 18:12 github-actions[bot]

[!WARNING]

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues. Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

mergify[bot] avatar Dec 22 '25 20:12 mergify[bot]

Dependency Review

The following issues were found:
  • βœ… 0 vulnerable package(s)
  • ❌ 3 package(s) with incompatible licenses
  • ❌ 1 package(s) with invalid SPDX license definitions
  • ⚠️ 7 package(s) with unknown licenses.
See the Details below.

License Issues

website/pnpm-lock.yaml

PackageVersionLicenseIssue Type
dompurify3.3.1(Apache-2.0 AND GPL-2.0-only AND MPL-2.0 AND MS-PL) OR (Apache-2.0 AND GPL-2.0-only AND MPL-2.0)Incompatible License
node-forge1.3.3(BSD-3-Clause AND GPL-1.0-or-later AND GPL-2.0 AND GPL-2.0-only) OR (BSD-3-Clause AND GPL-1.0-or-later AND GPL-2.0-only)Incompatible License
sax1.4.3BlueOak-1.0.0Incompatible License
posthog-js1.302.2SEE LICENSE IN LICENSEInvalid SPDX License
@csstools/postcss-position-area-property1.0.0NullUnknown License
@csstools/postcss-system-ui-font-family1.0.0NullUnknown License
@docsearch/core4.3.1NullUnknown License
@ai-sdk/gateway2.0.18NullUnknown License
@ai-sdk/react2.0.109NullUnknown License
@posthog/core1.7.1NullUnknown License
ai5.0.108NullUnknown License
Allowed Licenses: MIT, MIT-0, Apache-2.0, BSD-2-Clause, BSD-2-Clause-Views, BSD-3-Clause, ISC, MPL-2.0, 0BSD, Unlicense, CC0-1.0, CC-BY-3.0, CC-BY-4.0, CC-BY-SA-3.0, Python-2.0, OFL-1.1, LicenseRef-scancode-generic-cla, LicenseRef-scancode-unknown-license-reference, LicenseRef-scancode-unicode, LicenseRef-scancode-google-patent-license-golang

Scanned Files

  • website/pnpm-lock.yaml

github-actions[bot] avatar Dec 22 '25 20:12 github-actions[bot]