All expected policy violations are not displayed on every `pulumi preview/up` run.

[MitchellGerdisch]

What happened?

Ran pulumi preview (and up) against a local (and service-stored) policy-pack and noticed that not all expected policy violations are triggered on every run. Sometimes the expected policy violations are shown and sometimes none are shown and sometimes a subset of policy violations are shown.

Expected Behavior

All applicable policy violations should show on every pulumi preview (or up).

Steps to reproduce

1. Grab the project and policy-pack from: https://github.com/MitchellGerdisch/pulumi-work/tree/master/ts-policy-validate-resource-relationship.
2. cd policy-pack && npm i
3. cd ../pulumi-project && npm i
4. pulumi stack init dev
5. Run pulumi preview --policy-pack several times and note that sometimes you'll see all 3 expected policy violations and sometimes you'll see 1 or 2 or none of the policy violations.

NOTE: This same behavior happens if the policy-pack is published to an org or on pulumi up

Output of pulumi about

CLI
Version 3.55.0 Go Version go1.20 Go Compiler gc

Plugins NAME VERSION aws 5.30.0 awsx 1.0.2 docker 3.6.1 nodejs unknown

Host
OS darwin Version 12.5.1 Arch x86_64

Current Stack: xxxx/policy-project/dev

Found no resources associated with xxxx/dev

Found no pending operations associated with xxxx/dev

Backend
Name pulumi.com

Dependencies: NAME VERSION @types/node 16.18.14 @pulumi/aws 5.30.0 @pulumi/pulumi 3.56.0

Pulumi locates its logs in /var/folders/qp/6k0zsrj13rz5ll53hsmlksvw0000gq/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

Thanks for opening the issue, Mitch. I'll take a look.