aws-otel-lambda
aws-otel-lambda copied to clipboard
Published
20 hours ago •
aws-observability
aws-observability
ADOT Lambda Layer local testing
Is your feature request related to a problem? Please describe. AWS ADOT Lambda Python Sample app stuck till timeout in sam local invoke. Although the code in handler executed successfully, the ADOT Lambda Layer seems blocking the local invoke from exiting normally. This example works as expected when deployed to Lambda.
Describe the solution you'd like When Local testing, Function with ADOT Lambda layer should exit normally after execution
Describe alternatives you've considered Remove the ADOT layer when local testing, but this requires repeated manual work
Additional context
Steps to reproduce:
git clone [email protected]:aws-observability/aws-otel-lambda.git
cd aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam
sam build -u
sam local invoke --debug
Observed result:
Elysia % sam local invoke --region us-west-2 --debug
2024-05-28 18:38:58,817 | Using config file: samconfig.toml, config environment: default
2024-05-28 18:38:58,817 | Expand command line arguments to:
2024-05-28 18:38:58,818 | --template_file=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/ruojiazh/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2024-05-28 18:38:58,819 | local invoke command is called
2024-05-28 18:38:58,824 | No Parameters detected in the template
2024-05-28 18:38:58,862 | There is no customer defined id or cdk path defined for resource api, so we will use the resource logical id as the
resource id
2024-05-28 18:38:58,863 | Sam customer defined id is more priority than other IDs. Customer defined id for resource function is function
2024-05-28 18:38:58,864 | 0 stacks found in the template
2024-05-28 18:38:58,864 | No Parameters detected in the template
2024-05-28 18:38:58,880 | There is no customer defined id or cdk path defined for resource api, so we will use the resource logical id as the
resource id
2024-05-28 18:38:58,880 | Sam customer defined id is more priority than other IDs. Customer defined id for resource function is function
2024-05-28 18:38:58,881 | 2 resources found in the stack
2024-05-28 18:38:58,882 | Found Serverless function with name='function' and CodeUri='function'
2024-05-28 18:38:58,882 | --base-dir is not presented, adjusting uri function relative to
/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/template.yaml
2024-05-28 18:38:58,887 | Found one Lambda function with name 'function'
2024-05-28 18:38:58,888 | Invoking lambda_function.lambda_handler (python3.9)
2024-05-28 18:38:58,888 | Loading AWS credentials from session with profile 'None'
2024-05-28 18:38:59,227 | Resolving code path. Cwd=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build,
CodeUri=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function
2024-05-28 18:38:59,229 | Resolved absolute path to code is
/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function
2024-05-28 18:38:59,229 | Code /local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function is not a zip/jar
file
2024-05-28 18:38:59,230 | arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-24-0:1 is already cached. Skipping download
2024-05-28 18:38:59,636 | Local image is up-to-date
2024-05-28 18:38:59,640 | Checking free port on 127.0.0.1:8100
2024-05-28 18:38:59,642 | Using local image: samcli/lambda-python:3.9-x86_64-3260d442615df22e4be53571e.
2024-05-28 18:38:59,643 | Mounting /local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function as
/var/task:ro,delegated, inside runtime container
2024-05-28 18:39:00,014 | Starting a timer for 15 seconds for function 'function'
START RequestId: 18e3b183-d240-40f3-bc3c-2d660f6dea39 Version: $LATEST
{"level":"info","ts":1716921540.029972,"msg":"Launching OpenTelemetry Lambda extension","version":"v0.39.0"}
{"level":"info","ts":1716921540.0309246,"logger":"telemetryAPI.Listener","msg":"Listening for requests","address":":53612"}
{"level":"info","ts":1716921540.0309997,"logger":"telemetryAPI.Client","msg":"Subscribing","baseURL":"http://127.0.0.1:9001/2022-07-01/telemetry"}
{"level":"error","ts":1716921540.0312636,"logger":"telemetryAPI.Client","msg":"Subscription failed. Logs API is not supported! Is this extension running in a local sandbox?","status_code":202}
{"level":"info","ts":1716921540.0312836,"logger":"telemetryAPI.Client","msg":"Subscription success","response":"{\"errorMessage\":\"Telemetry API is not supported\",\"errorType\":\"Telemetry.NotSupported\"}\n"}
{"level":"info","ts":1716921540.0329914,"caller":"[email protected]/telemetry.go:55","msg":"Setting up own telemetry..."}
{"level":"info","ts":1716921540.0331674,"caller":"[email protected]/telemetry.go:97","msg":"Serving metrics","address":"localhost:8888","level":"Basic"}
{"level":"info","ts":1716921540.034368,"caller":"[email protected]/exporter.go:275","msg":"Deprecated component. Will be removed in future releases.","kind":"exporter","data_type":"metrics","name":"logging"}
{"level":"info","ts":1716921540.0594833,"caller":"[email protected]/service.go:143","msg":"Starting aws-otel-lambda...","Version":"v0.39.0","NumCPU":16}
{"level":"info","ts":1716921540.0595214,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1716921540.0596437,"caller":"[email protected]/otlp.go:102","msg":"Starting GRPC server","kind":"receiver","name":"otlp","data_type":"traces","endpoint":"localhost:4317"}
{"level":"info","ts":1716921540.0598555,"caller":"[email protected]/otlp.go:152","msg":"Starting HTTP server","kind":"receiver","name":"otlp","data_type":"traces","endpoint":"localhost:4318"}
{"level":"info","ts":1716921540.0599985,"caller":"[email protected]/service.go:169","msg":"Everything is ready. Begin running and processing data."}
{"level":"warn","ts":1716921540.0600317,"caller":"localhostgate/featuregate.go:63","msg":"The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.","feature gate ID":"component.UseLocalHostAsDefaultHost"}
MyBucket1
MyBucket2
-2024-05-28 18:39:15,017 | Function 'function' timed out after 15 seconds
-28 May 2024 18:39:15,028 [ERROR] (rapid) Invoke failed error=errResetReceived InvokeID=f3183651-cc6b-42fc-a050-cb60bc82b647
2024-05-28 18:39:15,736 | Container ID not defined, unable to fetch container state
2024-05-28 18:39:15,738 | Container was not created. Skipping deletion
2024-05-28 18:39:15,738 | Cleaning all decompressed code dirs
2024-05-28 18:39:15,739 | No response from invoke container for function
2024-05-28 18:39:15,739 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 18:39:15,747 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 18:39:15,748 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '064e6001-623a-48cc-bee3-ed8f62126453', 'installationId':
'f5068bb9-add9-4617-8f83-fdf7f78ec852', 'sessionId': '48d4dc59-f4e7-4716-b67b-fe983cedd9a1', 'executionEnvironment': 'CLI', 'ci': False,
'pyversion': '3.11.3', 'samcliVersion': '1.108.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName':
'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName':
'fcd52188567211bf1788ec5003f8a07ca1de3afc4605c8c2887fd8cb0d96f172', 'initialCommit': None}, 'duration': 16922, 'exitReason': 'success', 'exitCode':
0}}]}
2024-05-28 18:39:15,748 | Unable to find Click Context for getting session_id.
2024-05-28 18:39:15,752 | Sending Telemetry: {'metrics': [{'events': {'requestId': '66e243b6-0509-46c7-b171-f72bb048b391', 'installationId':
'f5068bb9-add9-4617-8f83-fdf7f78ec852', 'sessionId': '48d4dc59-f4e7-4716-b67b-fe983cedd9a1', 'executionEnvironment': 'CLI', 'ci': False,
'pyversion': '3.11.3', 'samcliVersion': '1.108.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '5a1d73ed92374980819d53bb918cc9a3', 'time_stamp': '2024-05-28 18:38:58.790',
'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '9fcd57b9cd85422aafc966b6d902cafd',
'time_stamp': '2024-05-28 18:38:58.817', 'exception_name': None}]}}}]}
2024-05-28 18:39:15,786 | Telemetry response: 200
2024-05-28 18:39:15,803 | Telemetry response: 200
Env
us-west-2
sam --info
{
"version": "1.108.0",
"system": {
"python": "3.11.3",
"os": "Linux-5.10.216-182.855.amzn2int.x86_64-x86_64-with-glibc2.26"
},
"additional_dependencies": {
"docker_engine": "25.0.3",
"aws_cdk": "2.134.0 (build 265d769)",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}```
