Documentation (examples): CDK example improvement with best practices

[niko-achilles]

Description of the improvement

Summary of the proposal

Provide examples that are structured in such a way that the application code (lambda handlers with aws power tools utilities) is separated from the stack deployment code.

Developers in order to organize their dependencies tend to compile application code with utilities as a process in the pipeline that executes on an other timepoint than the infrastructure relative code.

What is given ?

Given examples for cdk demonstrate the usage of AWS Lambda Powertools in TypeScript as utilities for Lambda functions and the examples guide the deployment of these Functions as part of cdk.

But the Functions rely on cdk for compilation.

Relative code in examples , as part of the stack code:

// infrastructure code applies the mechanisms as part of the infrastructure stack surface 
// by designating the construction of the function as stack resource with `tracingActive: true`

new ExampleFunction(this, 'MyFunction', {
      functionName: 'MyFunction',
      tracingActive: true,
    });

The application code on the other hand:

// uses the utilities of power tools
// imports the utilities like logger, tracer and metrics and uses them in application code

import { Metrics, MetricUnits } from '@aws-lambda-powertools/metrics';
import { Logger } from '@aws-lambda-powertools/logger';
import { Tracer } from '@aws-lambda-powertools/tracer';
....

How, where did you look for information

• Example web page:
• CDK examples repository:

Missing or unclear documentation

The examples make use of un-compiled Lambda functions , in pure typescript .ts files . and the examples use the NodejsFunction from library aws-cdk-lib/aws-lambda-nodejs.

The missing information is how to use the AWS Lambda Powertools as utilities for already compiled/transpiled Lambda function in .js files.

Improvement

use/choose from already created Lambda functions from the cdk examples which use the Lambda power tools utilities, create src directories for each of the chosen lambda functions with the typescript application code, compile application code and include in dist directories.

In addition, the application code, with src, dist directories that contain the application code lambda handlers with imported aws power tools utilities, should not be in the same directory as the cdk stack code.

This can / should be a good practice variant.

From a working project, I had a use case where my pipeline architecture is constructed in flexible way with stages and came to conclusion that the application code is kept separate from my infrastructure code.

Related existing documentation

yes , actually there is related notice that I found on terraform with cdk website Look for notice by following this link

The good part in those examples above is
that both kind of good practices are provided. One with precompiled typescript functions and the other one with pure typescript. The naming convention fully integrated for the examples with pure typescript is given. Hence the deployment stack takes the responsibility not only to deploy the application code , but also compile / transpile it.

Link compiled application code , separate from code infrastructure:

• https://github.com/hashicorp/learn-cdktf-assets-stacks-lambda

Link uncompiled application code, fully integrated with infrastructure code

• https://github.com/hashicorp/learn-cdktf-assets-stacks-lambda/tree/fully-integrated

Related issues, RFCs

Side note: From the web development practices developers compile in separate and in parallel utility specific code/dependencies . This is called in web development bundle splitting . Can have positive outcomes like performance in web, but in case of power tools and lambda i am not sure how those practices should be materialized or apply.
I use case can be applicable with lambda layers, where the layer is compiled separately and contains the utilities that can be used by application code.

[flochaz]

Thanks for this feedback and proposal, indeed we can do better here. We will look into it

[ijemmy]

@niko-achilles Let me make sure that I understand correctly.

You suggest to separate files that contain Lambda code into into a separated folder. The example is for cdktf. If we follow cdk official tutorial, I understand that they keep lambda files in a resources folder. So I assume that your proposal is something like this:

examples/cdk
  |- bin
  |- lib
    |- (cdk construct here)
  |- tests
  |- resources
    |- (lambda code here)

Is that correct?

ps. I will exclude bundle splitting or other optimization out of scope as it depends on the tool. In the future, we may add an example for other tools like SAM, Terraform, cdktf, or Serverless Framework. But the main purpose of example is just to illustrate how to integrate Powertools with the framework.

[dreamorosi]

To offer some additional context around the topic, the project structure used in these examples is in line with the reference project architecture suggested in the CDK v2 docs of the aws_lambda_nodejs construct:

.
├── lib
│   ├── my-construct.api.ts # Lambda handler for API
│   ├── my-construct.auth.ts # Lambda handler for Auth
│   └── my-construct.ts # CDK construct with two Lambda functions
├── package-lock.json # single lock file
├── package.json # CDK and runtime dependencies defined in a single package.json
└── tsconfig.json

Example copied example from page linked above

Breaking out each function in its own folder is suggested in those cases when functions don't share dependencies or when they are more complex than one single file. In this case the recommended project structure becomes:

.
├── packages
│   ├── cool-package
│   │   ├── lib
│   │   │   ├── cool-construct.api.ts
│   │   │   ├── cool-construct.auth.ts
│   │   │   └── cool-construct.ts
│   │   ├── package.json # CDK and runtime dependencies for cool-package
│   │   └── tsconfig.json
│   └── super-package
│       ├── lib
│       │   ├── super-construct.handler.ts
│       │   └── super-construct.ts
│       ├── package.json # CDK and runtime dependencies for super-package
│       └── tsconfig.json
├── package-lock.json # single lock file
├── package.json # root dependencies
└── tsconfig.json

Example copied example from page linked above

For the examples in this repo we have opted for the flat structure because all functions share most of the dependencies are are contained in a single file. Additionally, keeping this structure has allowed us to keep the CDK structure short since we are using the construct's defaults:

const fn = new NodejsFunction(this, functionName, {
  functionName: 'MyFunction',
  tracing: Tracing.ACTIVE,
});

For those who don't want to use this folder structure the NodejsFunction construct allows to specify entry and handler props:

new lambda.NodejsFunction(this, 'MyFunction', {
  entry: '/path/to/my/file.ts', // accepts .js, .jsx, .ts, .tsx and .mjs files
  handler: 'myExportedFunc', // defaults to 'handler'
});

Notably, the entry one accepts both TypeScript and Javascript functions, in addition to ESM ones.

With that said I'm open with clarifying the project structure choices in the README file of the CDK examples folder.

[niko-achilles]

@ijemmy exactly. Focus is to have application code with using powertools in a separate project folder outside the project folder that is specific for cdk infrastructure code .

As you wrote above:

... main purpose of example is just to illustrate how to integrate Powertools with the framework.

But why use the word resources for the application folder ? make it explicit like lambda-app or something that screams that it is actually an application. Inside there it should scream loud that the application code uses powertools, e.g package.json has dependencies powertools.

The cdk project should reference the transpiled application code by relative path reference for simplicity.

For transpilation of application code , use a simple method.

Is application code that uses powertools written in typescript ? then the fastest approach would be typescript compiler . I provide to you the config bases of typescript for information purposes. link .

I think the node14 is a good candidate: link as the aws lambda supports nodejs 14.

Is application code that uses powertools written in javascript ? then no transpilation for the purpose of example would be needed.

[niko-achilles]

A structure example in shake of simplicity. What do you think @dreamorosi ?

examples/cdk
  | - infrastructure
    |- bin
    |- lib
      |- (cdk construct here)
    |- tests
  |- lambda-application
    | - src
      | - lambda handler artifact with code that uses powertools
    | - dist // transpiled javascript 
      | - transpiled lambda handler artifact with powertools
    | - package.json // dependencies powertools

[dreamorosi]

A structure example in shake of simplicity. What do you think @dreamorosi ?

examples/cdk
  | - infrastructure
    |- bin
    |- lib
      |- (cdk construct here)
    |- tests
  |- lambda-application
    | - src
      | - lambda handler artifact with code that uses powertools
    | - dist // transpiled javascript 
      | - transpiled lambda handler artifact with powertools
    | - package.json // dependencies powertools

Could you please elaborate a bit more on what would be the benefit of this change and how it would benefit you and other customers using Powertools (& these examples)? I'm not sure I'm seeing the whole picture.

[niko-achilles]

@dreamorosi

the reasons of this issue / wish / improvement is described on my text when I opened this issue. In context of a second variant of best practice: separate application code from infrastructure code.

[ijemmy]

@niko-achilles

But why use the word resources for the application folder ?

Because the official tutorial uses that word. And I don't know if there is a global standard on naming best practices yet.

[ijemmy]

A structure example in shake of simplicity. What do you think @dreamorosi ?

examples/cdk
  | - infrastructure
    |- bin
    |- lib
      |- (cdk construct here)
    |- tests
  |- lambda-application
    | - src
      | - lambda handler artifact with code that uses powertools
    | - dist // transpiled javascript 
      | - transpiled lambda handler artifact with powertools
    | - package.json // dependencies powertools

This is clear to me now. Thanks!

[ijemmy]

Let me summarise this so we have actionable items

Reason for this change

1. We should separate application code from infrastructure code.
2. The example is hard to understand as it is. Readers must know:

• That functionName param is used (implicitly) to refer to the file name
• There are many files in a single folder and no logical grouping. Nothing indicate that the file is a lambda function

3. If we add examples for other tools (issue #385), we should reuse the Lambda code. Otherwise, we need to maintain 5 sets of Lambda functions (number of tools).

Reason of the current code structure

1. The example purpose is not to demonstrate best practices, but to show how to use Lambda Powertool. We just want to have a deployable example so users can experiment with it.
2. The example from @dreamorosi is an official CDK docs and the reference architecture for most CDK users. Nevertheless, this style is specific to aws_lambda_nodejs. The aws_lambda package example has a clearly named lambda-handler

Given that the maintainers need to focus on getting the tools production ready as soon as possible. This will be addressed after production release (Unless anyone wants to contribute!, please let me know). Let's tackle this with issue #385 so that the new structure we decide on is future-proof for other tools (as the way they build application code are different).

[ijemmy]

One more thing, I prefer that we don't keep transpiled JS files in Git repo.

[niko-achilles]

One more thing, I prefer that we don't keep transpiled JS files in Git repo.

the transpiled files not. But the mechansims to build in a folder like dist and the command to build the application code should be in the examples, so that when i download the code i can execute the command , see the build in the dist folder. The deployment infrastructure code should reference the files inside the dist folder.

[saragerion]

I agree with what @ijemmy said here.

Thanks @niko-achilles for opening this issue.

[niko-achilles]

i also can derive from my code base, some examples that use powertools with javascript (with JsDoc magic) and typescript. When you decide on the structure i can contribute , ping me in case you need some support , respecting the time to release a production ready lib of powertools.

// cc. @ijemmy , @dreamorosi @saragerion

[niko-achilles]

Looking at this issue , i have pushed in my repository an example of Lambda written in Javascript with Typescript support in VSCode. Link: https://github.com/niko-achilles/lambda-powertools-js-example

The lambda function uses the tracer and commons module (see side note) .

Given my motivation for this issue was to learn from examples of lambda functions and decide which tool to use , e.g SAM or CDK or terraform , will the structure of examples in the repository of powertools change, so that i can leverage the ability to choose tools for aws infrastructure ? Examples Link: https://github.com/awslabs/aws-lambda-powertools-typescript/tree/main/examples

side note: i have added support for IntelliSense in Javascript with named import because commons module exports CustomEvent type without a named export-- i could have used require functionality, but for playing i decided to not to use require functionality. Link commons module: https://github.com/awslabs/aws-lambda-powertools-typescript/blob/main/packages/commons/src/index.ts#L5

Link Intellisense support of my example: https://github.com/niko-achilles/lambda-powertools-js-example/blob/main/app/src/types/%40aws-lambda-powertools/commons/index.d.ts

// cc. @dreamorosi

[niko-achilles]

@dreamorosi have updated my example with points of interest. Deployed in the aws cloud to test the runtime behavior and developer experience. In the example some concepts may inspire people for examples improvement in the powertools project.

Given my motivation for this issue was to learn from examples of lambda functions and decide which tool to use , e.g SAM or CDK or terraform , will the structure of examples in the repository of powertools change, so that i can leverage the ability to choose tools for aws infrastructure ?

As in the example provided in my personal repo i have a project structure which is different than powertools.

I mean the current project structure of examples in powertools can be improved.

In case aware of improvement this Issue can be closed . @ijemmy @flochaz @saragerion

[dreamorosi]

Hi @niko-achilles, thanks for sharing your sample repo and sorry for not getting back to you on the previous comment.

Updating the contents of the examples folder is still on our backlog, and while I'm not yet sure about how it will look like exactly, we are going to separate the infrastructure code from the functions one - similar to your example.

At the moment we have another contributor updating the SAM examples (#1140) and the functions code. Once that activity is complete we'll do the same with the CDK ones and re-use the same code in both examples.

I'd like to keep this issue open as a reminder of updating the CDK examples.

[github-actions[bot]]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.