Python: CdkFunction.environment is dereferenced prematurely instead of returning IResolvable

Open object-Object opened this issue 2 years ago • 1 comments

Describe the bug

CdkFunction.environment is a lazily evaluated value which can be resolved to CfnFunction.EnvironmentProperty.

In TypeScript and Go, environment returns LazyAny and typeregistry.anonymousProxy respectively, both of which can then be resolved with stack.resolve(environment).

In Python, environment returns an actual EnvironmentProperty instead of a reference, with variables=None. This resolves to an empty dict.

Expected Behavior

I expected CfnFunction.environment to be an IResolvable value which could then be resolved with stack.resolve(environment).

Current Behavior

CfnFunction.environment is an empty EnvironmentProperty value with no reference, so it's not possible to get the environment variables from this value.

JSII_DEBUG output from Go:

> {"api":"get","property":"environment","objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"}}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
[@jsii/kernel] serialize LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
} {
  serializationClass: 'Struct',
  typeRef: {
    type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
    optional: true
  }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
  '$jsii.byref': 'Object@10047',
  '$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}

From Python:

> {"objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"},"property":"environment","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
[@jsii/kernel] serialize LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
} {
  serializationClass: 'Struct',
  typeRef: {
    type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
    optional: true
  }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
  '$jsii.byref': 'Object@10047',
  '$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}
> {"objref":{"$jsii.byref":"Object@10047"},"property":"variables","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'Object@10047' } variables
[@jsii/kernel] value: undefined
[@jsii/kernel] serialize undefined {
  serializationClass: 'Map',
  typeRef: { type: { collection: [Object] }, optional: true }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] ret: undefined
< {"ok":{}}

Note the extra request at the end of the Python logs. This is from the exact same line of code, cfnFunction.Environment() and cfn_function.environment respectively.

Reproduction Steps

Paste each block into the stack file in the project created by cdk init for that language, then run cdk ls.

Python:

from aws_cdk import Stack, aws_lambda as lambda_
from constructs import Construct

class CdkPyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        func = lambda_.Function(
            self,
            "Function",
            code=lambda_.Code.from_inline("_"),
            runtime=lambda_.Runtime.PYTHON_3_10,
            handler="_",
            environment={"KEY": "value"},
        )

        cfn_function = func.node.default_child
        assert isinstance(cfn_function, lambda_.CfnFunction)
        environment = cfn_function.environment
        print(environment)
        print(self.resolve(environment))

Output:

EnvironmentProperty()
{}
CdkPyStack

TypeScript (working, for comparison):

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class CdkTsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    let func = new lambda.Function(this, "Function", {
      code: lambda.Code.fromInline("_"),
      runtime: lambda.Runtime.PYTHON_3_10,
      handler: "_",
      environment: { KEY: "value" },
    });

    let cfnFunction = func.node.defaultChild as lambda.CfnFunction;
    let environment = cfnFunction.environment;
    console.log(environment);
    console.log(this.resolve(environment));
  }
}

Output:

LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
{ variables: { KEY: 'value' } }
CdkTsStack

Possible Solution

Here's what I've been able to figure out by stepping through with a debugger. Hopefully this helps.

CfnFunction.environment is a property which calls jsii.get(), aka kernel.get(). https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_kernel/init.py#L358-L366

The value returned from kernel.get() is an ObjRef, which is passed to _reference_map.resolve_reference() via the @_dereferenced decorator and _recursize_dereference(). https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_kernel/init.py#L133-L151

The ref is something like Object@10047, so class_fqn is Object. The condition on line 104-106 below evaluates to True and it ends up creating and returning an EnvironmentProperty value with no data (because it hasn't been resolved with CDK yet). https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_reference_map.py#L102-L128

Additional Information/Context

My use case is to create a custom aspect which adds default environment variables to all Lambdas in a stack. The aspect shouldn't overwrite variables set directly on a function, so it needs to know which ones already exist, if any.

SDK version used

JSII 1.84.0, CDK lib and cli 2.84.0, Constructs 10.2.55

Environment details (OS name and version, etc.)

Windows 10.0.14393 Build 14393, Python 3.11.3

Jun 20 '23 20:06 object-Object

Update: it's possible to work around this issue by calling internal JSII functions with the below code, but it feels fragile.

This works by essentially intercepting the output of jsii.get() before it goes through _recursize_dereference(), changing the type of the reference to IResolvable, and then dereferencing it.

from jsii._kernel.types import GetRequest, ObjRef
from jsii._reference_map import InterfaceDynamicProxy, resolve_reference
from aws_cdk.aws_lambda import CdkFunction

obj: CdkFunction = ...

response = jsii.kernel.provider.get(
    GetRequest(objref=obj.__jsii_ref__, property="environment")
).value

return resolve_reference(jsii.kernel, ObjRef(response.ref, ["aws-cdk-lib.IResolvable"]))

Aug 29 '23 20:08 object-Object