Provide a way to opt-out of secretness for Python dynamic providers

[justinvp]

The serialized Python dynamic provider code is marked as a secret by default with https://github.com/pulumi/pulumi/pull/13315. This was done to ensure any credentials used by a dynamic provider would not be stored in plaintext in the state. However, this can lead to performance issues due to https://github.com/pulumi/pulumi/issues/15538.

We should consider providing a way to opt-out of secretness, or even consider changing the default back and making it opt-in. Perhaps we could come up with a way to detect secretness similar to what the Node.js SDK does for its dynamic provider implementation.

In the meantime, if you know your Python dynamic provider implementation does not capture any secrets, and you'd like to opt-out of secretness, you could use a transformation to do so as a workaround. For example, calling register_stack_transformation before creating any of the dynamic resources:

import pulumi

def unsecret_dynamic_providers(args: pulumi.ResourceTransformationArgs):
    if args.type_ == "pulumi-python:dynamic:Resource":
        args.props["__provider"] = pulumi.Output.unsecret(args.props["__provider"])
        return pulumi.ResourceTransformationResult(args.props, args.opts)
    return None

pulumi.runtime.register_stack_transformation(unsecret_dynamic_providers)

[AndriiP-AIT]

@justinvp, do you know of any existing workaround to avoid updating already existing dynamic providers (other than manually patching local sources)? In my opinion, the change from PR 13315, where _provider became encrypted by default, broke backward compatibility (without a major version change).

The issue is that dynamic providers created with Pulumi Python versions earlier than 3.75.0 will be recreated if the Pulumi module is updated to version 3.75.0 or later. Consequently, dependent resources will also be recreated, which is not always possible.

It looks like your PR 15864 would address exactly what we need. Are there any plans to merge it? 🙂

[pulumi-bot]

This issue has been addressed in PR #15864 and shipped in release v3.128.0.