Bug: Layers fail to deploy correctly

[sambler]

Description:

When defining a layer in the template and using a makefile, the source dir is copied instead of the artifacts dir.

This changed recently, I think with the 1.110 update.

I am seeing this using the lambda python3.9 runtime.

Steps to reproduce:

Define layer in template

Resources:
  mnsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: MyNotesLayer
      Description: Common modules used by functions.
      ContentUri: mnsLayer
      CompatibleRuntimes:
        - python3.9
      RetentionPolicy: Delete
    Metadata:
      BuildMethod: makefile

Makefile-

PYVERS = 3.9
PROJDIR = $(HOME)/work/mytheta.dev/note_site/api
MNSLAYERDIR = $(PROJDIR)/mnsLayer

build-mnsLayer:
	mkdir -p "$(ARTIFACTS_DIR)/python"
	cd "$(MNSLAYERDIR)"
	pip-${PYVERS} install --quiet --target="$(ARTIFACTS_DIR)/python" --upgrade -r requirements.txt
	cp $(MNSLAYERDIR)/utils.py $(ARTIFACTS_DIR)/python/utils.py
	cd $(ARTIFACTS_DIR)/python && python${PYVERS} -m compileall -o 2 -q .

Observed result:

The source dir (mnsLayer) contains Makefile, requirements.txt and utils.py - these three files are found in the lambda runtime as /opt/Makefile, /opt/requirements.txt and /opt/utils.py

I can adjust my source to use the utils module with

import sys
sys.path.insert(0, '/opt')
import utils

but the extra modules in requirements are not found in the lambda runtime.

Locally, .aws-sam/build/mnsLayer/python contains the expected files, but they are not found in the lambda runtime.

Expected result:

Previously this setup was installing utils.py so that it could be imported with import utils as well as extra modules in requirements being available.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: FreeBSD 13.2
2. sam --version: 1.110.0
3. AWS region: us-east-1

{
  "version": "1.110.0",
  "system": {
    "python": "3.9.16",
    "os": "FreeBSD-13.2-STABLE-amd64-64bit-ELF"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "2.123.0 (build a594112)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

the portion of sam build --debug output that contains the layer build

2024-03-06 20:27:44,476 | Same Layer build definition found, adding layer (Previous: LayerBuildDefinition(mnsLayer,     
/home/shane/work/mytheta.dev/note_site/api/mnsLayer, , 7781afc6-72c3-4a6f-a3e9-5d0cacb5b85a, makefile, ['python3.9'],   
x86_64, {}), Current: LayerBuildDefinition(mnsLayer, /home/shane/work/mytheta.dev/note_site/api/mnsLayer, ,             
748c9402-9a93-4d64-ad00-ad858ec8af9f, makefile, ['python3.9'], x86_64, {}), Layer:                                      
<samcli.lib.providers.provider.LayerVersion object at 0x8853b2bb0>)                                                     
2024-03-06 20:27:44,483 | Building layer 'mnsLayer'                                                                     
2024-03-06 20:27:44,485 | Loading workflow module 'aws_lambda_builders.workflows'                                       
2024-03-06 20:27:44,489 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided',     
dependency_manager=None, application_framework=None)'                                                                   
2024-03-06 20:27:44,493 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', 
dependency_manager='cli-package', application_framework=None)'                                                          
2024-03-06 20:27:44,496 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go',            
dependency_manager='modules', application_framework=None)'                                                              
2024-03-06 20:27:44,501 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java',        
dependency_manager='gradle', application_framework=None)'                                                               
2024-03-06 20:27:44,505 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java',         
dependency_manager='maven', application_framework=None)'                                                                
2024-03-06 20:27:44,508 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs',        
dependency_manager='npm', application_framework=None)'                                                                  
2024-03-06 20:27:44,512 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', 
dependency_manager='npm-esbuild', application_framework=None)'                                                          
2024-03-06 20:27:44,517 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python',        
dependency_manager='pip', application_framework=None)'                                                                  
2024-03-06 20:27:44,520 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby',        
dependency_manager='bundler', application_framework=None)'                                                              
2024-03-06 20:27:44,524 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust',    
dependency_manager='cargo', application_framework=None)'                                                                
2024-03-06 20:27:44,526 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided',   
dependency_manager=None, application_framework=None)'                                                                   
2024-03-06 20:27:44,528 | Running workflow 'CustomMakeBuilder'                                                          
2024-03-06 20:27:44,530 | mnsLayer: Running CustomMakeBuilder:CopySource                                                
2024-03-06 20:27:44,532 | Copying source file (/home/shane/work/mytheta.dev/note_site/api/mnsLayer/Makefile) to         
destination (/tmp/tmpgeyred0a/Makefile)                                                                                 
2024-03-06 20:27:44,534 | Creating target folders at /tmp/tmpgeyred0a/__pycache__                                       
2024-03-06 20:27:44,536 | Copying directory metadata from source                                                        
(/home/shane/work/mytheta.dev/note_site/api/mnsLayer/__pycache__) to destination (/tmp/tmpgeyred0a/__pycache__)         
2024-03-06 20:27:44,538 | Copying source file                                                                           
(/home/shane/work/mytheta.dev/note_site/api/mnsLayer/__pycache__/utils.cpython-39.pyc) to destination                   
(/tmp/tmpgeyred0a/__pycache__/utils.cpython-39.pyc)                                                                     
2024-03-06 20:27:44,540 | Copying source file (/home/shane/work/mytheta.dev/note_site/api/mnsLayer/utils.py) to         
destination (/tmp/tmpgeyred0a/utils.py)                                                                                 
2024-03-06 20:27:44,542 | Copying source file (/home/shane/work/mytheta.dev/note_site/api/mnsLayer/requirements.txt) to 
destination (/tmp/tmpgeyred0a/requirements.txt)                                                                         
2024-03-06 20:27:44,544 | CustomMakeBuilder:CopySource succeeded                                                        
2024-03-06 20:27:44,545 | mnsLayer: Running CustomMakeBuilder:MakeBuild                                                 
2024-03-06 20:27:44,547 | mnsLayer: Current Artifacts Directory :                                                       
/home/shane/work/mytheta.dev/note_site/api/.aws-sam/build/mnsLayer                                                      
2024-03-06 20:27:44,548 | executing Make: ['make', '--makefile',                                                        
'/home/shane/work/mytheta.dev/note_site/api/mnsLayer/Makefile', 'build-mnsLayer']                                       
`/home/shane/work/mytheta.dev/note_site/api/mnsLayer/Makefile' is up to date.
mkdir -p "/home/shane/work/mytheta.dev/note_site/api/.aws-sam/build/mnsLayer/python"
cd "/home/shane/work/mytheta.dev/note_site/api/mnsLayer"
pip-3.9 install --quiet --target="/home/shane/work/mytheta.dev/note_site/api/.aws-sam/build/mnsLayer/python" --upgrade -r requirements.txt
cp /home/shane/work/mytheta.dev/note_site/api/mnsLayer/utils.py /home/shane/work/mytheta.dev/note_site/api/.aws-sam/build/mnsLayer/python/utils.py
cd /home/shane/work/mytheta.dev/note_site/api/.aws-sam/build/mnsLayer/python && python3.9 -m compileall -o 2 -q .
2024-03-06 20:27:59,245 | CustomMakeBuilder:MakeBuild succeeded                                                         

[sidhujus]

Hi @sambler Thanks for reporting the issue. Could you try building with the --use-container flag and then deploying? I was able to access the dependencies installed through the layer when I did that. Additionally, do you know which version of SAM CLI last worked as you expected with the layers?

Meanwhile ill bring this up with the team to determine what we should be doing here

[sambler]

I don't have a linux setup to run docker, so --use-container fails.

Just looked at my snapshots to check versions from a month ago

aws_sam_cli - 1.109.0 - now 1.110.0 aws_sam_translator - 1.84.0 - now 1.85.0 aws_lambda_builders - 1.45.0 - now 1.46.0 boto3_stubs - 1.34.32 - now 1.34.43 cfn_lint - 0.85.1 - now 0.85.2 ruamel.yaml - 0.18.5 - now 0.18.6

[varadkarpe]

I faced the same issue when I executed sam deploy with --template-file and --config-file options. Upon downloading the contents of the Lambda layer, all I found was the source directory and not the artifacts directory. But when I deployed without the --template-file and --config-file options the deployment was successful and the contents of the lambda layer had the artifacts directory in it. In my case, this is not an issue which has occurred after a version update since this is a completely new deployment. Environment: python3.9 aws-sam-cli: v1.118.0

A colleague of mine faced the same problem with v1.120

[YisusCristH4K3r]

Can confirm what @varadkarpe said, the deploy seems to only package the ContentUri source contents into the layer if the --template-file option is passed. Without it, it seems like the proper built artifact is deployed. Just to add some more context, using sam build both with and without the --template-file option produce the correct artifact output for the layer inside the .aws-sam/build/ folder. aws-sam-cli: 1.124.0 Env: python3.10