Run all examples touched by a pr as ci tests

[marcindulak]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

It's a heads up.

1. As reported in several places (e.g. https://github.com/pulumi/examples/issues/395 or https://github.com/pulumi/examples/issues/666 or https://github.com/pulumi/examples/issues/1057) not all examples are tested by github actions. Some documentation examples break and are reported by end-users, but the number of unreported, broken examples is unknown and circulates as a word of mouth (e.g. https://github.com/pulumi/pulumi-azure-native/issues/1495#issuecomment-1171523665). Sometimes broken examples are being disabled and may not come back soon (e.g. https://github.com/pulumi/pulumi-azure-native/issues/898 or https://github.com/pulumi/pulumi-azure-native/pull/1599 or https://github.com/pulumi/pulumi-azure-native/issues/304 or https://github.com/pulumi/examples/issues/1188).

2. The newest successful status of "Run Examples Tests From PR" github action workflow appears 25 days ago and the last successful "Smoke Test Specific Version of CLI" two months ago.

3. Pull requests with a failed status of "Run Examples Tests From PR" are merged. They may fail on unrelated tests, but don't actually test the focus feature of the PR (let's take for example https://github.com/pulumi/examples/pull/1234).

4. The tests output a red "Error" when the run appears successful (folllowing the example of https://github.com/pulumi/examples/pull/1234 above)

5. The native providers have limited coverage of resource tests. The usage of the native examples is not clear: on one hand they are tested, but on the other hand not recommended for end-users https://github.com/pulumi/pulumi-azure-native/issues/1043#issuecomment-899854236.

Affected area/feature

https://github.com/pulumi/examples, but also other repos that contain examples.