nuclei icon indicating copy to clipboard operation
nuclei copied to clipboard

feat(templating): add vars templating into yaml inputs (ytt)

Open alban-stourbe-wmx opened this issue 10 months ago • 3 comments

Proposed changes

Checklist

  • [X] Pull request is created against the dev branch
  • [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • [X] I have added tests that prove my fix is effective or that my feature works
  • [X] I have added necessary documentation (if appropriate)

Summary by CodeRabbit

  • New Features
    • Added a command-line flag to enable text templating for variables in YAML input files.
    • Introduced support for processing YAML templates using the ytt templating engine.
  • Bug Fixes
    • Enhanced error handling and debug logging during YAML parsing.
  • Tests
    • Added tests to verify YAML parsing with variable text templating enabled.
  • Chores
    • Updated dependencies and Go module versions.

alban-stourbe-wmx avatar Jun 12 '25 13:06 alban-stourbe-wmx

Walkthrough

Adds Carvel ytt-based variable templating for YAML input: new CLI flags and Options fields, propagation through input provider, ytt integration utilities, YAML parser templating step, tests, and dependency updates.

Changes

Cohort / File(s) Change Summary
CLI & config
cmd/nuclei/main.go
Added flags --vars-text-templating (-vtt) and --var-file-paths (-vfp) and YAML config decoding for vars.
Options / types
pkg/types/types.go
Added public fields VarsTextTemplating and VarsFilePaths and updated Copy() to propagate them.
Input format options
pkg/input/formats/formats.go
Added VarsTextTemplating bool and VarsFilePaths []string to InputFormatOptions.
Provider wiring
pkg/input/provider/interface.go
Propagated the new templating fields into input provider initialization.
YAML parsing & templating
pkg/input/formats/yaml/multidoc.go
Optional pre-processing with ytt when templating enabled; adjusted decoding/error handling and debug logging.
ytt integration
pkg/input/formats/yaml/ytt.go
New helpers to run Carvel ytt programmatically (templates, data-values, var files, noop UI).
Tests
pkg/input/formats/yaml/multidoc_test.go
Added TestYamlFormatterParseWithVariables exercising ytt templating end-to-end.
Module deps
go.mod
Updated Go toolchain directive and dependencies; added carvel.dev/ytt and related packages; several dependency version changes.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant User
    participant CLI
    participant ConfigLoader
    participant InputProvider
    participant YamlMultiDocFormat
    participant yttEngine

    User->>CLI: invoke with flags (--vars-text-templating / --var-file-paths)
    CLI->>ConfigLoader: parse flags & config file
    ConfigLoader->>InputProvider: initialize with Options (includes templating fields)
    InputProvider->>YamlMultiDocFormat: Parse(input)
    alt VarsTextTemplating enabled
        YamlMultiDocFormat->>yttEngine: render templates (templates, data-values, var files)
        yttEngine-->>YamlMultiDocFormat: rendered YAML
    end
    YamlMultiDocFormat->>YamlMultiDocFormat: decode YAML docs -> raw requests
    YamlMultiDocFormat-->>InputProvider: return parsed requests

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

  • dogancanbakir
  • ehsandeep

Pre-merge checks (3 passed)

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title "feat(templating): add vars templating into yaml inputs (ytt)" succinctly and accurately captures the primary change—adding ytt-based variable templating for YAML input files—using conventional commit style; it is specific, concise, and directly related to the changes in the PR.
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.

Poem

In a burrow of bytes the rabbit did say,
"I'll ytt-mix your YAML and hop it away."
Vars stitched like carrots in rows neat and tidy,
Tests hum a chorus — the output looks mighty.
🐇✨ Hop, template, and play!

[!TIP]

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

  • Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
  • Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.

✨ Finishing touches
  • [ ] 📝 Generate Docstrings
🧪 Generate unit tests
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Jun 12 '25 13:06 coderabbitai[bot]

heyyyy 🙋 @dogancanbakir @tarunKoyalwar @ehsandeep

alban-stourbe-wmx avatar Jun 12 '25 13:06 alban-stourbe-wmx

that are amazing features well done @alban-stourbe-wmx !!!

sacha-athias-wmx avatar Jun 12 '25 13:06 sacha-athias-wmx

I've added the ability to populate a yaml file containing variables directly from a yaml file. Why? Because it's possible to inject data structures directly into ytt, whereas via the var cli you can only inject strings.

As shown in the example above, this makes it possible to create queries that mix the encoding of the form body with other data structures (sending a json to a form variable), or even to inject arrays.

Command to test it: nuclei -config pkg/input/formats/testdata/ytt/ytt-profile.yaml -proxy http://127.0.0.1:8080 -v

Screenshot 2025-06-24 at 18 48 44

alban-stourbe-wmx avatar Jun 24 '25 16:06 alban-stourbe-wmx

fix: #6260

alban-stourbe-wmx avatar Jul 03 '25 12:07 alban-stourbe-wmx

I've fixed a few issues with tests and Request parsing. Very interesting addition - Thanks for the PR!

Thanks ! ;) Do you know if this feature will be included in the next version? @Mzack9999

alban-stourbe-wmx avatar Jul 08 '25 12:07 alban-stourbe-wmx

@alban-stourbe-wmx I'm merging in dev, it will be available in next release!

Mzack9999 avatar Sep 11 '25 19:09 Mzack9999