Introduce examples cache

[t0yv0]

Rolling out the support for the external pulumi TF converter has slowed down iterating in the repo significantly. It's at a point where I find it extremely difficult to iterate.

make tfgen: 4:26-5:00 make build_python: 5:00

And make build_sdks is 5 min per language.

We should wait until the next bridge release to merge this but I wanted to see if the cache helps. It seems like it does and it brings make tfgen down to 1:00, and so does make build_python etc. This is better. It also has potential to speed up CI a bit if we add cache handling to GHA.

Some remaining small opportunities:

• it's not caching rendering failures so continues to invoke external pulumi for them; this could be cached also
• cache key computation is very conservative and sensitive to go.mod etc changes; somewhat necessarily so, interesting to see how it works out in practice, will most of the jobs we do be a cache miss?
• I'd really like to decouple building from generating as I most cases I'm just concerned with generating the SDKs; dotnet and java builds in particular take forever
• If we could share developer and CI caches we could profit, not obvious how though

[github-actions[bot]]

Does the PR have any schema changes?

Does the PR have any schema changes?

Looking good! No breaking changes found. No new resources/functions.

Maintainer note: consult the runbook for dealing with any breaking changes.

[iwahbe]

cache key computation is very conservative and sensitive to go.mod etc changes; somewhat necessarily so, interesting to see how it works out in practice, will most of the jobs we do be a cache miss?

We don't see a change to go.{mod,sum} when building against replace, so I hope that we will see cache hits on local builds.

I actually think caching in CI will be a major win here. Each SDK build in CI runs 2-3 times: once in run-acceptence-tests.yml, another time in main.yml and finally in release.yml (if that commit is to be released). It is very common that CI builds the same SDK set 3, from identical code.

[VenelinMartinov]

Very exciting!

[t0yv0]

Yeah that's the plan:

It is very common that CI builds the same SDK set 3, from identical code.

[t0yv0]

We don't see a change to go.{mod,sum} when building against replace, so I hope that we will see cache hits on local builds.

That gets me thinking. If you put a replace in go.mod that'll change the hash of go.mod. If you have a go.work at the right place it will also change the hash if go.work itself changes but it does not chase it into the actual version pointed to by go.work:

https://github.com/pulumi/pulumi-terraform-bridge/blob/master/pkg/tfgen/examples_cache.go#L158

We straddle a fine line here, the misfortunate I would really like to avoid is failing to ship example corrections by inadvertently serving a stale cached version of the example; and secondarily to that, having situations where local builds disagree from CI because of the cache. We might need to iterate on what's part of the cache key a bit and the conventions here.

[iwahbe]

We straddle a fine line here, the misfortunate I would really like to avoid is failing to ship example corrections by inadvertently serving a stale cached version of the example; and secondarily to that, having situations where local builds disagree from CI because of the cache. We might need to iterate on what's part of the cache key a bit and the conventions here.

For example, AFAIK the cache key doesn't include resources.go, which I know contributes to codegen outputs. I think this will take a couple of careful iterations here, but that the end result will be worth it.

[github-actions[bot]]

Is README.md missing any configuration options?

assumeRoleWithWebIdentity not found in Configuration section customCaBundle not found in Configuration section defaultTags not found in Configuration section ec2MetadataServiceEndpoint not found in Configuration section ec2MetadataServiceEndpointMode not found in Configuration section httpProxy not found in Configuration section httpsProxy not found in Configuration section ignoreTags not found in Configuration section noProxy not found in Configuration section retryMode not found in Configuration section s3UsEast1RegionalEndpoint not found in Configuration section sharedConfigFiles not found in Configuration section skipRequestingAccountId not found in Configuration section stsRegion not found in Configuration section tokenBucketRateLimiterCapacity not found in Configuration section useDualstackEndpoint not found in Configuration section

Please add a description for each of these options to README.md. Details about them can be found in either the upstream docs or schema.json.

[t0yv0]

Closing, this has been upstreamed to ci-mgmt and the bridge.