TestExamples/ts-nlb-simple/upgrade-preview-only fails on CI

[t0yv0]

What happened?

There is an issue with a replay-based provider upgrade test. It seems to pass for me locally but fails in CI. The failure reads something like this:

warning: using pulumi-resource-awsx from $PATH at /home/runner/work/pulumi-awsx/pulumi-awsx/bin/pulumi-resource-awsx
  warning: using pulumi-resource-awsx from $PATH at /home/runner/work/pulumi-awsx/pulumi-awsx/bin/pulumi-resource-awsx
    pulumi:pulumi:Stack: (same)
      [urn=urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:pulumi:Stack::ts-nlb-simple-p-it-fv-az491-5-ts-nlb-sim-b0cccd6e]
  warning: using pulumi-resource-awsx from $PATH at /home/runner/work/pulumi-awsx/pulumi-awsx/bin/pulumi-resource-awsx
          +-aws:lb/loadBalancer:LoadBalancer: (replace)
              [id=arn:aws:elasticloadbalancing:us-west-2:616138583583:loadbalancer/net/nginx-lb-c00afb4/821fc191ad125fbe]
              [urn=urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::awsx:lb:NetworkLoadBalancer$aws:lb/loadBalancer:LoadBalancer::nginx-lb]
              [provider: urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_5_35_0::fcb88445-5634-43ad-9dd9-d9d032832b45 => urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_6_5_0::output<string>]
            ~ subnets: [
                ~ [0]: "subnet-0016572b" => "subnet-03711d3b9b21b3a8e"
                ~ [1]: "subnet-d7e7fe9c" => "subnet-06e8296c053e2b952"
                ~ [2]: "subnet-c7d926bf" => "subnet-0fc2dc8f8ba906919"
                ~ [3]: "subnet-43f43a1e" => "subnet-037f366816336db85"
              ]
          +-aws:lb/targetGroup:TargetGroup: (replace)
              [id=arn:aws:elasticloadbalancing:us-west-2:616138583583:targetgroup/nginx-lb-3cfc5c4/4f00a96b0fd82d3b]
              [urn=urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::awsx:lb:NetworkLoadBalancer$aws:lb/targetGroup:TargetGroup::nginx-lb]
              [provider: urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_5_35_0::fcb88445-5634-43ad-9dd9-d9d032832b45 => urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_6_5_0::output<string>]
            ~ vpcId: "vpc-4b82e033" => "vpc-043be661de8760bec"
          +-aws:lb/listener:Listener: (replace)
              [id=arn:aws:elasticloadbalancing:us-west-2:616138583583:listener/net/nginx-lb-c00afb4/821fc191ad125fbe/8f5ee5491d839947]
              [urn=urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::awsx:lb:NetworkLoadBalancer$aws:lb/listener:Listener::nginx-lb-0]
              [provider: urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_5_35_0::fcb88445-5634-43ad-9dd9-d9d032832b45 => urn:pulumi:p-it-fv-az491-5-ts-nlb-sim-b0cccd6e::ts-nlb-simple::pulumi:providers:aws::default_6_5_0::output<string>]
            ~ defaultActions : [
                ~ [0]: {
                        ~ targetGroupArn: "arn:aws:elasticloadbalancing:us-west-2:616138583583:targetgroup/nginx-lb-3cfc5c4/4f00a96b0fd82d3b" => output<string>
                      }
              ]
            ~ loadBalancerArn: "arn:aws:elasticloadbalancing:us-west-2:616138583583:loadbalancer/net/nginx-lb-c00afb4/821fc191ad125fbe" => output<string>
  Resources:
      +-3 to replace
      2 unchanged

Basically there is an unexpected replace plan on this simple program:

import * as awsx from "@pulumi/awsx";

// // Create a network load balancer.
const lb = new awsx.lb.NetworkLoadBalancer("nginx-lb");

Example

import * as awsx from "@pulumi/awsx";

// // Create a network load balancer.
const lb = new awsx.lb.NetworkLoadBalancer("nginx-lb");

Output of pulumi about

Locally I'm on 3.86.0.

Additional context

Could it be an issue with the program behaving differently on Mac OS vs Linux?

Could it be due to a Pulumi version difference?

Since the upgrade testing framework is very new, also cannot rule out some misconfigured test aspect.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[danielrbradley]

This is actually caused by the lb default behavour trying to use the private subnets from the default subnet in the current account. However, the account changes between recording locally and running in CI. We should avoid using default VPCs in upgrade tests and also avoid ever using invokes which return information related to the current environment.

[t0yv0]

Ah that's a good note. Super tricky. Perhaps we could avoid this just by being really systematic about only ever recording on CI (not locally), but I guess then again these recordings will not be useful for fast-testing locally. Very annoying.