aws-cdk
aws-cdk copied to clipboard
aws
feat(lambda-python-alpha): cache python lambda dependencies usig lambda layer
Issue # (if applicable)
Closes #23829
Reason for this change
cdk synth takes very long to synthesize a PythonFunction construct, because the dependencies are getting installed regardless if there was any changes made to them.
The dependencies, whether they are specified in a requirements.txt file, a poetry.lock file or a pipenv.lock file, get installed as part of the CMD in the bundling phase, meaning that we don't use docker's cache or any other caching mechanism and re-install the dependencies from the internet every time.
Trying to compute a custom assetHash based on the dependencies file won't work, because of where the call for install is currently placed (in the CMD of the bundling phase container)
This causes deployment times to rise significantly, in some cases from minutes to hours.
Description of changes
The PythonFunction construct in function.ts will introduce a new prop: layer which is defined as:
/**
* Whether or not to create a layer for the function's dependencies.
* @default - No layer is created.
*/
readonly layer?: boolean;
If layer is true, we create the layer before the constructor of PythonFunction is called, and attach the layer to PythonFunction in the constructor's layers prop.
To control whether dependencies are installed during the Bundling phase, a new prop is introduced to BundlingOptions and is defined as:
/**
* Whether or not to install the dependencies
* @default true
*/
readonly installDependencies?: boolean;
Since bundling is used for the PythonLayerVersion and PythonFunction props, this prop defaults to true, to not alter any behavior of PythonLayerVersion.
When initializing a PythonFunction, if layer is true, and we are creating a layer, the installDependencies is set to true for the layer creation, and false for the constructor of PythonFunction, so dependencies will only be installed in the layer.
To compute the assetHash for the layer, I added a dependenciesHash(entry) method inside the Packaging class, with this we can compute the hash of the lock file, and in the created layer we can use AssetHashType.CUSTOM with this hash.
bundling: {
...props.bundling,
installDependencies: true,
// assetExcludes: ["TODO: exclude everything except the dependencies file"]
assetHashType: AssetHashType.CUSTOM,
assetHash: Packaging.dependenciesHash(entry),
},
Description of how you validated changes
This is my first PR here, and in open source in general, I am working on adding and validating the changes with unit tests, I would appreciate some help with that, if possible.
Checklist
- [ ] My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES
TODOs
- assetExcludes for the created layer, exclude all files besides the lock file
- Add tests
- Edit
README.md
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
The pull request linter fails with the following errors:
❌ Features must contain a change to a README file.
❌ Features must contain a change to a test file.
❌ Features must contain a change to an integration test file and the resulting snapshot.
PRs must pass status checks before we can provide a meaningful review.
If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.
AWS CodeBuild CI Report
- CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
- Commit ID: 9891f5bab3254f7d330e2d76ef9693e37fd6483e
- Result: FAILED
- Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
Exemption Request
This is a first PR for me in this codebase, i have the general idea of the design and the required code changes, and could use some help with the PR process
This PR has been in the CHANGES REQUESTED state for 3 weeks, and looks abandoned. To keep this PR from being closed, please continue work on it. If not, it will automatically be closed in a week.
This PR has been deemed to be abandoned, and will be automatically closed. Please create a new PR for these changes if you think this decision has been made in error.
The pull request linter fails with the following errors:
❌ Features must contain a change to a README file.
❌ Features must contain a change to a test file.
❌ Features must contain a change to an integration test file and the resulting snapshot.
PRs must pass status checks before we can provide a meaningful review.
If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.
✅ A exemption request has been requested. Please wait for a maintainer's review.