Enable registry to source metadata from pulumi registry APIs

[sean1588]

Enable registry to source metadata from registry APIs.

Part of: https://github.com/pulumi/pulumi-service/issues/26019

Enables resourcedocsgen to source metadata from package endpoints. Will follow up with pulling schema from endpoints once available.

This PR enables resourcedocsgen to pull package metadata from the registry apis instead of sourcing from the data files in the registry repo.

There is a boolean flag, --use-api, I added to enable building from the APIs. By default it will continue to use the filesystem. This will allow us to rollback easily if needed. We can flip this and test it out once the registry service contains the packages. Currently I am seeing an empty list of packages being returned, so I am assuming this has yet to be done. So far I am "guessing" what would be returned just based on inspecting the code in pulumi service. We can wait to get this in until then.

For right now, we should expect the same end result with these changes as it is still configured to use the metadata files on the file system. See previews for confirmation.

[github-actions[bot]]

Your site preview for commit 9b9e03bb is ready! :tada:

http://registry--origin-pr-7057-9b9e03bb.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit 058d69bc is ready! :tada:

http://registry--origin-pr-7057-058d69bc.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit ffecacb1 is ready! :tada:

http://registry--origin-pr-7057-ffecacb1.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit d14a56cb is ready! :tada:

http://registry--origin-pr-7057-d14a56cb.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit 8f3c1b99 is ready! :tada:

http://registry--origin-pr-7057-8f3c1b99.s3-website.us-west-2.amazonaws.com/registry.

[flostadler]

@sean1588 Can you sync with @iwahbe on how we plan to publish to the registry service?

While your change adds the necessary bits to pull from the registry service, Ian is working on the publishing side (which will still be driven by pulumi/registry). Given that the two are heavily intertwined we should align the implementation as well.

[github-actions[bot]]

Your site preview for commit 2d10a9d4 is ready! :tada:

http://registry--origin-pr-7057-2d10a9d4.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit dfffeb30 is ready! :tada:

http://registry--origin-pr-7057-dfffeb30.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit 2fc5728b is ready! :tada:

http://registry--origin-pr-7057-2fc5728b.s3-website.us-west-2.amazonaws.com/registry.

[sean1588]

@iwahbe I have addressed your comments.

[github-actions[bot]]

Your site preview for commit 41d81ffe is ready! :tada:

http://registry--origin-pr-7057-41d81ffe.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit c8245ab7 is ready! :tada:

http://registry--origin-pr-7057-c8245ab7.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit 4df7085b is ready! :tada:

http://registry--origin-pr-7057-4df7085b.s3-website.us-west-2.amazonaws.com/registry.

[sean1588]

@iwahbe tests added!

[github-actions[bot]]

Your site preview for commit 02e88271 is ready! :tada:

http://registry--origin-pr-7057-02e88271.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit 315dae9e is ready! :tada:

http://registry--origin-pr-7057-315dae9e.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit de25c1d9 is ready! :tada:

http://registry--origin-pr-7057-de25c1d9.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit a4729205 is ready! :tada:

http://registry--origin-pr-7057-a4729205.s3-website.us-west-2.amazonaws.com/registry.

[iwahbe]

We should be able to run a live test of this now. The live registry should have all the providers.

[github-actions[bot]]

Your site preview for commit def43427 is ready! :tada:

http://registry--origin-pr-7057-def43427.s3-website.us-west-2.amazonaws.com/registry.

[github-actions[bot]]

Your site preview for commit e8dac290 is ready! :tada:

http://registry--origin-pr-7057-e8dac290.s3-website.us-west-2.amazonaws.com/registry.