Pulumi plan causes a violation when dynamic credentials are resolved during preview operations

[stazz]

What happened?

The docker-build provider crashed on up command with --plan=<path> argument with the following message:

error: resource urn:pulumi:<stack>::<project>::docker-build:index:Image::<resource> violates plan: properties changed: ~~registries[{[{map[address:{&{{<ecr URL>}}} password:{&{{<token1>}}} username:{&{{<username>}}}]}]}!={[{map[address:{&{{<ecr URL>}}} password:{&{{<token2>}}} username:{&{{<username>}}}]}]}]

Example

The code which causes this is like this:

import * as pulumi from "@pulumi/pulumi";
import * as legacyEcr from "@pulumi/aws/ecr";
import * as ecr from "@pulumi/aws-native/ecr";
import * as docker from "@pulumi/docker-build";
import * as process from "node:process";
import path from "node:path";

const repo = new ecr.Repository("...", {
  repositoryName: "...",
  ...
 });
const buildDirectory = path.join(process.cwd(), "docker");

const token = legacyEcr.getAuthorizationTokenOutput();

export const image = new docker.Image("...", {
  // Why is "push" and "exports" like this? See: https://github.com/pulumi/pulumi-docker-build/issues/252#issuecomment-2340554610
  push: false,
  exports: [
    {
      registry: {},
    },
  ],
  buildOnPreview: true, // Always build on preview
  tags: [pulumi.interpolate`${repo.repositoryUri}:latest`],
  dockerfile: {
    location: path.join(buildDirectory, "Dockerfile"),
  },
  context: {
    location: buildDirectory,
  },
  buildArgs: {
    ...
  },
  platforms: [docker.Platform.Linux_arm64],
  registries: [
    {
      address: pulumi.secret(repo.repositoryUri),
      username: pulumi.secret(token.userName),
      password: token.password,
    },
  ],
});

It works fine without --plan functionality, but not without. The plan was also extremely simple - on preview it said there are no changes to the stack (as it should be, since I changed nothing). However, on up command, the error message was seen (which btw printed the username and password in plaintext to console output), and process crashed.

Output of pulumi about

CLI
Version      3.134.1
Go Version   go1.23.1
Go Compiler  gc

Host
OS       debian
Version  12.7
Arch     aarch64

Backend
Name           pulumi.com
URL            https://app.pulumi.com
User           Unknown
Organizations
Token type     personal

Pulumi locates its logs in /tmp by default
warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from <cwd>). If you have not created a project yet, use `pulumi new` to do so: no project file found
warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from <cwd>). If you have not created a project yet, use `pulumi new` to do so: no project file found

The output is a bit wonky since I am utilising Pulumi Automation API to run my pipeline, as it is a bit complex (has some other components than just Pulumi), and I am storing the state and encryption key in AWS (not using Pulumi cloud). Furthermore, I am running Pulumi inside Docker container pulumi/pulumi-nodejs-22:3.134.1.

But the versions in the package.json are like this:

    "@pulumi/aws-native": "0.125.0",
    "@pulumi/aws": "6.53.0",
    "@pulumi/docker-build": "0.0.6",
    "@pulumi/postgresql": "3.12.0",
    "@pulumi/pulumi": "3.134.1",
    "@pulumi/random": "4.16.6",

Additional context

Now that I've written the full report, I am not sure whether this actually belongs here or in AWS provider. Maybe the AWS provider for some reason provides same token during preview (and even refresh, as I've run that as well), but during up it is not? Since in the error message, the password values DO differ. I don't know. Feel free to move the issue over the AWS provider repo if it really belongs there. 👍

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[stazz]

Okay, I found also a somewhat heavyweight workaround which is not optimal at all, but it DOES prevent crashing. I've added registries to the ignoreChanges provider options of docker.Image resource, and now up with no-op plan passes. This might bite me back later, but for now, it at least prevents crashing, and makes using plan files in general feasible.

export const image = new docker.Image(
  "...",
  {
    ...
  },
  {
    ignoreChanges: ["registries"],
  },
); 

[rquitales]

@stazz Thanks for highlighting the limitation with Pulumi plans and dynamic tokens causing a plan violation. As noted in the plan documentation, Pulumi tries to resolve all outputs during Preview and writes them into the plan output. Since the ECR token can be resolved during Preview (assuming the ECR repo is already created), Pulumi will always write the token to the plan output. This is problematic because it leads to a failure during the subsequent pulumi up due to token rotation.

I agree that using ignoreChanges to bypass this isn't the best approach. A better solution would be marking the token as unknown during Preview, preventing Pulumi from including it in the local plan output.

Here's an example of how to modify the token constant:

const token = pulumi.runtime.isDryRun()
  ? { userName: pulumi.unknown, password: pulumi.unknown }
  : legacyEcr.getAuthorizationTokenOutput();

This ensures the dynamic credentials won't be stored in the plan output.

[rquitales]

Transferring this issue to pulumi/pulumi as it relates to Pulumi plans with dynamic credentials being resolved during Preview and stored.

[stazz]

@rquitales Thanks for commenting and providing a workaround, greatly appreciated! 👍 I will give it a try soon, but I am sure it will work nicely.

[jwilliams-fn]

I just attempted this workaround with some deploy-time annotations but I still experience the error. I'm trying to add a few deployment annotations (deploy user, deploy date, commit hash, etc) to kubernetes resources during a stack transformation:

DEPLOY_DATE = pulumi.UNKNOWN if pulumi.runtime.is_dry_run() else datetime.datetime.now().replace(microsecond=0).isoformat()
deployment_annotations = { ... }
args.props["metadata"]["annotations"] = deployment_annotations

return pulumi.ResourceTransformResult(
    props=args.props,
    opts=args.opts,
)

[bcheidemann]

I have hit a similar issue with the AWS provider.

For context: during local runs, the AWS profile is used, while in CI runs the access key, secret key and token are used.

function getProvider(
  name: string,
  args: GetProviderArgs,
  opts?: pulumi.ResourceOptions,
) {
  if (process.env.CI === "true") {
    return new aws.Provider(
      name,
      {
        ...args,
        profile: undefined,
        accessKey: process.env[`${args.credentialsPrefix}AWS_ACCESS_KEY_ID`],
        secretKey:
          process.env[`${args.credentialsPrefix}AWS_SECRET_ACCESS_KEY`],
        token: process.env[`${args.credentialsPrefix}AWS_SESSION_TOKEN`],
      },
      opts,
    );
  }
  return new aws.Provider(
    name,
    {
      ...args,
      accessKey: undefined,
      secretKey: undefined,
      token: undefined,
    },
    opts,
  );
}

I have hit two problems (one of which is related to this issue and the other if solved would likely also solve the other):

1. Noisy diff (the -profile,+accessKey,+secretKey,+token diff is just noise as it doesn't represent any actual changes to my infra. I can't ignore changes to these values as this causes the deployment to fail. This is not ideal but I can live with the noisy diff as I almost always deploy from CI.
2. When I output a plan.json in CI, this is manually reviewed. A separate workflow then executes the plan, but this issues new credentials, causing a plan violation.

I think a solution for 1 would most likely solve 2 as well.

I tried applying the solution suggested by @rquitales, but unfortunately that results in a very noisy diff due to the following (and similar):

@ previewing update.....
 ~  aws:backup:Vault [REDACTED] [diff: -__defaults,forceDestroy,region,tagsAll]; warning: The provider for this resource has inputs that are not known during preview.
...
Diagnostics:
  aws:backup:Selection ([REDACTED]):
    warning: The provider for this resource has inputs that are not known during preview.
    This preview may not correctly represent the changes that will be applied during an update.
...

Indeed, the preview does "not correctly represent the changes that will be applied".

As far as I can tell, there doesn't seem to be any viable way to use plans when:

1. The deployment must run in CI
2. Short lived credentials are used with the AWS provider
3. Manual review of the plan is required (which presumably is the primary purpose of creating a plan)

Are there any workarounds for this?