Improve the diff display in the Terraform provider's output when using `keep_original_source = true`

[aballiet]

Expected Behavior

We expect to have a proper diff when applying changes to the content of a flow.

Actual Behaviour

Currently output is unreadable.

Steps To Reproduce

1. Create a flow, like this one :

resource "kestra_flow" "example" {
  namespace            = "dev"
  flow_id              = "my-flow"
  keep_original_source = true
  content              = <<EOT
id: "my-flow"
namespace: "dev"
inputs:
  - name: my-value
    type: STRING

variables:
  first: "2"

tasks:
  - id: t2
    type: io.kestra.core.tasks.debugs.Echo
    format: first {{task.id}}
    level: TRACE

taskDefaults:
  - type: io.kestra.core.tasks.debugs.Echo
    values:
      format: third {{flow.id}}

EOT
}

1. Apply your flow (create it on your Kestra instance)
2. Change whatever row
3. terraform plan and check diff
4. You should have something like this :

You can reproduce the experience with keep_original_source = false. In that case we will work as expected.

Environment Information

• Kestra Version: 0.12 SNPASHOT
• Operating System (OS / Docker / Kubernetes): k8s
• Java Version (If not docker):

Example flow

No response

[aballiet]

Here is a loom video showcasing the issue: https://www.loom.com/share/c498cb2085e542c89070b00e067a0c44?sid=dbd0ea7c-a50c-4887-a64d-20b55c7f7f7c

Please ensure it is resolved before removing the option keep_original_source = false

[aballiet]

After digging a bit with @tchiotludo, this issue should be fixed by following using the CustomizeDiff function in the Terraform SDK to normalize and compare YAML content. This approach ensures consistent key ordering and structure during diff computations.

Here’s how you can implement this:

• Parse and Normalize YAML: Use a YAML parsing library (e.g., gopkg.in/yaml.v3 or similar) to parse the provided content string into a structured representation (like a map). Then, serialize it back to YAML or JSON with consistent key ordering.
• Implement CustomizeDiff: Read the content attribute from the planned resource configuration. Normalize the YAML content. Compare it against the current state to determine if changes exist.

We can leverage setAttribute function to do it, see doc here

ChatGPT code:

package provider

import (
	"bytes"
	"github.com/hashicorp/terraform-plugin-framework/datasource"
	"github.com/hashicorp/terraform-plugin-framework/resource"
	"github.com/hashicorp/terraform-plugin-framework/types"
	"gopkg.in/yaml.v3"
	"sort"
)

func normalizeYAML(input string) (string, error) {
	var parsed map[string]interface{}

	// Parse YAML
	if err := yaml.Unmarshal([]byte(input), &parsed); err != nil {
		return "", err
	}

	// Sort and serialize
	normalized, err := marshalSorted(parsed)
	if err != nil {
		return "", err
	}

	return normalized, nil
}

func marshalSorted(data interface{}) (string, error) {
	var buffer bytes.Buffer

	// Custom sort implementation for YAML keys
	encoder := yaml.NewEncoder(&buffer)
	defer encoder.Close()

	if err := encodeWithSortedKeys(encoder, data); err != nil {
		return "", err
	}

	return buffer.String(), nil
}

func encodeWithSortedKeys(encoder *yaml.Encoder, data interface{}) error {
	switch v := data.(type) {
	case map[string]interface{}:
		keys := make([]string, 0, len(v))
		for k := range v {
			keys = append(keys, k)
		}
		sort.Strings(keys)

		ordered := make(map[string]interface{})
		for _, k := range keys {
			ordered[k] = v[k]
		}
		return encoder.Encode(ordered)

	case []interface{}:
		for _, item := range v {
			if err := encodeWithSortedKeys(encoder, item); err != nil {
				return err
			}
		}
	default:
		return encoder.Encode(v)
	}
	return nil
}

func CustomizeDiff(ctx resource.CustomizeDiffContext, req resource.CustomizeDiffRequest, resp *resource.CustomizeDiffResponse) {
	var content types.String

	// Read the "content" attribute
	req.Config.GetAttribute(ctx, path.Root("content"), &content)
	if content.IsNull() || content.IsUnknown() {
		return
	}

	normalized, err := normalizeYAML(content.ValueString())
	if err != nil {
		resp.Diagnostics.AddError("YAML Normalization Error", "Failed to normalize YAML: "+err.Error())
		return
	}

	// Store normalized content as a computed attribute or compare it for diffs
	resp.Plan.SetAttribute(ctx, path.Root("normalized_content"), types.StringValue(normalized))
}

[anna-geller]

Ben, FYI the customer-request label was added because LMFR was also asking for this. Marking as P1 so we can handle it in the next cycle

[aballiet]

Ben, FYI the customer-request label was added because LMFR was also asking for this. Marking as P1 so we can handle it in the next cycle

That would be awesome 🤗

[aballiet]

After updating to Kestra v0.21.12, have this new diff:

-> beforeCommands need to be pass with '["pip install pandas", ...]'

[anna-geller]

@fdelbrayelle this issue is open for quite a while, would be cool if you could investigate in the upcoming cycle 🙏

[fdelbrayelle]

According to @AcevedoR, let's wait for @Skraye next week to sync about this one since there would be some deprecated things to check before. This keep_original_source field may be not kept after all.

[tchiotludo]

Everything should keep only the yaml endpoint, and we can remove all the json endpoint for sure, no more customer are using version where the yaml endpoint didn't exists. so keep_original_source could be remove at first (using only yaml)

[aballiet]

Everything should keep only the yaml endpoint, and we can remove all the json endpoint for sure, no more customer are using version where the yaml endpoint didn't exists. so keep_original_source could be remove at first (using only yaml)

Would this fix the problem with diff display in Terraform ?

[AcevedoR]

yes!