Extract test.go files from published packages

[blampe]

Description

We have a lot of dependencies brought in by codegen due to some files named test.go. These are included in published packages and compiled artifacts because Go only excludes files ending with _test.go from downstream dependencies.

AFAICT these files are intended for test helpers, but they bloat our consumers by pulling in things like AWS and Azure SDKs. These dependencies then make their way into bridged providers via tfgen's codegen imports:

❯ go mod why github.com/aws/aws-sdk-go-v2/service/s3
# github.com/aws/aws-sdk-go-v2/service/s3
github.com/pulumi/pulumi-docker/provider/v4/cmd/pulumi-tfgen-docker
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen
github.com/pulumi/pulumi/pkg/v3/codegen/dotnet
github.com/pulumi/pulumi/pkg/v3/testing/integration        <- problematic 
github.com/pulumi/pulumi/pkg/v3/resource/stack
github.com/pulumi/pulumi/pkg/v3/secrets/cloud
github.com/pulumi/pulumi/pkg/v3/authhelpers
gocloud.dev/blob
gocloud.dev/blob.test
gocloud.dev/blob/s3blob
github.com/aws/aws-sdk-go-v2/service/s3

This PR re-arranges GenerateProgramBatchTest logic to consolidate everything under pkg/codegen/testing/test instead of the language-specific codegen packages. By doing so we no longer include test dependencies in non-test packages that are consumed downstream.

As an example, this takes the pulumi-language-go binary from ~61MB down to ~36MB, and speeds up build times:

go clean -cache && time go build .
(master) go build .  81.06s user 17.36s system 470% cpu 20.933 total
(PR)     go build .  47.77s user 9.20s system 530% cpu 10.737 total

This doesn't completely remove these dependencies from downstream because most also include pkg/v3/testing/integration in tests. It does only compile these dependencies into the test binary, though.

Before:

❯ go mod why github.com/aws/aws-sdk-go-v2/service/sts
# github.com/aws/aws-sdk-go-v2/service/sts
github.com/pulumi/pulumi-random/provider/v4/cmd/pulumi-tfgen-random
github.com/pulumi/pulumi-terraform-bridge/pf/tfgen
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen
github.com/pulumi/pulumi/pkg/v3/codegen/dotnet
github.com/pulumi/pulumi/pkg/v3/testing/integration
github.com/pulumi/pulumi/pkg/v3/resource/stack
github.com/pulumi/pulumi/pkg/v3/secrets/cloud
github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test
github.com/aws/aws-sdk-go-v2/service/sts

After:

# github.com/aws/aws-sdk-go-v2/service/sts
github.com/pulumi/pulumi-random/provider/v4
github.com/pulumi/pulumi-random/provider/v4.test     <- expected 
github.com/pulumi/providertest
github.com/pulumi/pulumi/pkg/v3/testing/integration
github.com/pulumi/pulumi/pkg/v3/resource/stack
github.com/pulumi/pulumi/pkg/v3/secrets/cloud
github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test
github.com/aws/aws-sdk-go-v2/service/sts

NB: This is on top of v3.113.0 because v3.113.3 introduced some API changes to codegen which haven't been picked up by the bridge yet.

The awsx tests are failing with make: *** No rule to make target 'upstream'. and I'm not sure what to do with that.

Checklist

• [ ] I have run make tidy to update any new dependencies
• [ ] I have run make lint to verify my code passes the lint check
  • [ ] I have formatted my code using gofumpt

• [ ] I have added tests that prove my fix is effective or that my feature works

• [ ] I have run make changelog and committed the changelog/pending/<file> documenting my change

• [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

[pulumi-bot]

Changelog

[uncommitted] (2024-04-24)

Bug Fixes

• [pkg] Remove test dependencies from codegen packages #16011