aws-otel-lambda icon indicating copy to clipboard operation
aws-otel-lambda copied to clipboard

Published 20 hours ago • verified publisher icon aws-observability

publish Docker images for multi-stage builds to use

Open smcoll opened this issue 3 years ago • 10 comments

i've had difficulty adding X-Ray tracing for a python application using OpenTelemetry with a container-based Lambda. The published extension isn't immediately applicable to that use case.

Ideally, there'd be images published to ECR which could be used in multi-stage builds something like this (maybe per python version?):

FROM public.ecr.aws/lambda/aws-otel-lambda:latest as otel-base

FROM public.ecr.aws/lambda/python:3.8 
COPY src/ $LAMBDA_TASK_ROOT
COPY --from otel-base /opt/ /opt/

That may not work for all use cases, but could at least work for images based on public.ecr.aws/lambda/python.

i understand if that's outside the purpose of this project.

i've tried doing something like in a Dockerfile myself, and i'm getting Lambda timeouts whenever the xray exporter is included in the config (using logging exorter only works fine), with hardly any context to go on.

Maybe i could get a little help:

FROM public.ecr.aws/lambda/python:3.8 as python-base
# [... some ENV set up here]


FROM python-base as otel-base

RUN yum -y install git  make

RUN wget https://go.dev/dl/go1.17.3.linux-amd64.tar.gz && rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.3.linux-amd64.tar.gz

ENV PATH=$PATH:/usr/local/go/bin

RUN git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git

WORKDIR $LAMBDA_TASK_ROOT/aws-otel-lambda

# Patch ADOT
RUN ./patch-upstream.sh

# instead of running python/build.sh, which is creating zip archives, run the underlying build.sh scripts.
# opentelemetry-lambda/python/src/build.sh requires docker, so use Make instead. we also make an empty collector_build dir so the script doesn't fail; it's looking for the collector we are adding later
RUN cd opentelemetry-lambda/python/src/otel && \
    mkdir build && \
    mkdir collector_build && \
    touch collector_build/tmp && \
    ARTIFACTS_DIR=build make build-OTelLayer && \
    mv build/python /opt/

# FIXME: /opt/otel-instrument is in /opt/python/otel-instrument?

# Build collector
RUN cd opentelemetry-lambda/collector && make build && cp -r build/extensions /opt

RUN mkdir /opt/collector-config && cp opentelemetry-lambda/collector/config* /opt/collector-config/

# use our defined collector configs
COPY "./applications/collector-config/" /opt/collector-config/

FROM python-base as application

# [... add and install python libs]

COPY --from=otel-base /opt/ /opt/

Does the instrumentor need to be installed in the same environment as the application's python libs? In my case, i've been using poetry to install to /var/lang/lib/python3.8/.

smcoll avatar Dec 07 '21 21:12 smcoll

Hi @smcoll,

I think having container images for the Lambda layers is a perfectly valid feature request, so I can mark this as such.

Have you referenced the build script for our managed layer to see how to use otel-instrument script? You can find it here: https://github.com/aws-observability/aws-otel-lambda/blob/main/python/build.sh

willarmiros avatar Dec 21 '21 00:12 willarmiros

I feel like this similar issue I answered a few days ago is related, so just dropping the link here: https://github.com/aws-observability/aws-otel-community/issues/37#issuecomment-998284294

Regarding this in your code:

FIXME: /opt/otel-instrument is in /opt/python/otel-instrument?

The upstream opentelemetry-lambda repo actually moves the python/otel-instrument file to be at the otel-instrument path when it creates the zip containing the build so it _should_ be in /opt/otel-instrument`: https://github.com/open-telemetry/opentelemetry-lambda/blob/8e77f5b6a4dbc2665bc15cf829cafa700456b1e9/python/src/otel/Dockerfile#L17

We run this Docker Image in this aws-otel-lambda repo here:

https://github.com/aws-observability/aws-otel-lambda/blob/486911eb8e05f4512d4abafaca0b9430914394e5/python/build.sh#L9-L12

One last complexity is that we rename it and replace it with our own "otel-instrument" because we want to add AWS X-Ray Python configuration values:

https://github.com/aws-observability/aws-otel-lambda/blob/486911eb8e05f4512d4abafaca0b9430914394e5/python/build.sh#L29-L31

If you just run ./python/build.sh from this aws-otel-lambda repo it would do all these steps, but if you are doing your own version of python/build.sh these are things to keep in mind to make sure your final file directory looks like the same like the zip file the original buid.sh generates 🙂

NathanielRN avatar Dec 27 '21 23:12 NathanielRN

@smcoll wondering if you got this working with python lambdas deployed as containers. Currently struggling through the same issue.

brett-bim avatar Jan 28 '22 18:01 brett-bim

Getting back around to this...

It does look like copying files from the published layer could do the trick (although that requires passing through AWS creds into the build), but i'm using Python 3.9 and there isn't a layer published for it yet: https://github.com/aws-observability/aws-otel-lambda/issues/222.

Next approach was to use the build scripts as-is in a container so i could use the layer artifacts with COPY FROM, but this requires docker-in-docker, or equivalent. Using Amazon Linux 2022: docker run --rm -it public.ecr.aws/amazonlinux/amazonlinux:2022 bash:

bash-5.1# cd /home
bash-5.1# yum -y install git wget tar zip unzip make docker pip
bash-5.1# wget https://go.dev/dl/go1.17.3.linux-amd64.tar.gz && rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.3.linux-amd64.tar.gz
bash-5.1# export PATH=$PATH:/usr/local/go/bin
bash-5.1# systemctl enable docker
bash-5.1# systemctl start docker
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
bash-5.1# 
bash-5.1# # at this point i realize i need docker-in-docker to do this... if that were configured, i'd do the following:
bash-5.1# git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git
bash-5.1# cd aws-otel-lambda/python
bash-5.1# ./build.sh

In the end, i just want appropriate files to put in the /opt dir in my Lambda container, based on the python:3.9 image. So i investigated the build script to better understand what is happening here. Maybe i can use a few different docker stages each with a different base image and COPY FROM to the the result it need?

Is this explicitly building for 3.8, and if so, how can i adjust to build for 3.9?

Stepping through the build script, to validate my comprehension:

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L3-L7

This section says it's building the collector, but it appears to me that it results in a no-op; i ran them inside a container (without || exit) after checking out this source code, and i see it overlays a dir from the upstream submodule and creates python/build/ and contents, but then after popd i don't see any remaining artifacts... are we looking for a collector binary here?

(Details) 
bash-4.2# git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git
Cloning into 'aws-otel-lambda'...
remote: Enumerating objects: 1794, done.
remote: Counting objects: 100% (180/180), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 1794 (delta 75), reused 99 (delta 34), pack-reused 1614
Receiving objects: 100% (1794/1794), 1.75 MiB | 9.85 MiB/s, done.
Resolving deltas: 100% (908/908), done.
Submodule 'opentelemetry-lambda' (https://github.com/open-telemetry/opentelemetry-lambda.git) registered for path 'opentelemetry-lambda'
Cloning into '/var/task/aws-otel-lambda/opentelemetry-lambda'...
remote: Enumerating objects: 1922, done.        
remote: Counting objects: 100% (680/680), done.        
remote: Compressing objects: 100% (238/238), done.        
remote: Total 1922 (delta 495), reused 466 (delta 421), pack-reused 1242        
Receiving objects: 100% (1922/1922), 591.42 KiB | 5.28 MiB/s, done.
Resolving deltas: 100% (857/857), done.
Submodule path 'opentelemetry-lambda': checked out 'e13a8ce7328e6739ed510d66bc5461427fee8eb2'
bash-4.2# cd aws-otel-lambda/
bash-4.2# ls -al
total 100
drwxr-xr-x 12 root root  4096 Mar  1 21:20 .
drwxr-xr-x  1 root root  4096 Mar  1 21:20 ..
drwxr-xr-x  5 root root  4096 Mar  1 21:20 adot
-rw-r--r--  1 root root  7850 Mar  1 21:20 CHANGELOG.md
-rw-r--r--  1 root root   309 Mar  1 21:20 CODE_OF_CONDUCT.md
-rw-r--r--  1 root root  3160 Mar  1 21:20 CONTRIBUTING.md
drwxr-xr-x  4 root root  4096 Mar  1 21:20 dotnet
drwxr-xr-x  9 root root  4096 Mar  1 21:20 .git
-rw-r--r--  1 root root    85 Mar  1 21:20 .gitattributes
drwxr-xr-x  4 root root  4096 Mar  1 21:20 .github
-rw-r--r--  1 root root   127 Mar  1 21:20 .gitignore
-rw-r--r--  1 root root   130 Mar  1 21:20 .gitmodules
drwxr-xr-x  4 root root  4096 Mar  1 21:20 go
drwxr-xr-x  7 root root  4096 Mar  1 21:20 java
-rw-r--r--  1 root root 10142 Mar  1 21:20 LICENSE
drwxr-xr-x  6 root root  4096 Mar  1 21:20 nodejs
-rw-r--r--  1 root root    67 Mar  1 21:20 NOTICE
drwxr-xr-x 11 root root  4096 Mar  1 21:20 opentelemetry-lambda
-rwxr-xr-x  1 root root  1626 Mar  1 21:20 patch-upstream.sh
drwxr-xr-x  5 root root  4096 Mar  1 21:20 python
-rw-r--r--  1 root root  2467 Mar  1 21:20 README.md
drwxr-xr-x  3 root root  4096 Mar  1 21:20 sample-apps
bash-4.2# cd python/
bash-4.2# ls -al
total 24
drwxr-xr-x  5 root root 4096 Mar  1 21:20 .
drwxr-xr-x 12 root root 4096 Mar  1 21:20 ..
-rwxr-xr-x  1 root root  758 Mar  1 21:20 build.sh
drwxr-xr-x  3 root root 4096 Mar  1 21:20 integration-tests
drwxr-xr-x  3 root root 4096 Mar  1 21:20 sample-apps
drwxr-xr-x  2 root root 4096 Mar  1 21:20 scripts
bash-4.2# pushd ../opentelemetry-lambda/collector
/var/task/aws-otel-lambda/opentelemetry-lambda/collector /var/task/aws-otel-lambda/python
bash-4.2# ls -al 
total 268
drwxr-xr-x  4 root root   4096 Mar  1 21:20 .
drwxr-xr-x 11 root root   4096 Mar  1 21:20 ..
-rw-r--r--  1 root root   2736 Mar  1 21:20 collector.go
-rw-r--r--  1 root root    251 Mar  1 21:20 config.yaml
drwxr-xr-x  2 root root   4096 Mar  1 21:20 extension
-rw-r--r--  1 root root   4964 Mar  1 21:20 go.mod
-rw-r--r--  1 root root 219786 Mar  1 21:20 go.sum
drwxr-xr-x  2 root root   4096 Mar  1 21:20 lambdacomponents
-rw-r--r--  1 root root   2358 Mar  1 21:20 main.go
-rw-r--r--  1 root root   1624 Mar  1 21:20 Makefile
-rw-r--r--  1 root root   1042 Mar  1 21:20 print.go
-rw-r--r--  1 root root   3222 Mar  1 21:20 README.md
-rw-r--r--  1 root root      7 Mar  1 21:20 VERSION
bash-4.2# make package
rm -rf build/
Building otel collector extension
mkdir -p /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/extensions
GOOS=linux GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath -ldflags "-s -w -X main.GitHash=e13a8ce7328e6739ed510d66bc5461427fee8eb2 -X main.Version=v0.1.0 -X github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter.collectorDistribution=opentelemetry-collector-lambda" -o /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/extensions .
Package zip file for collector extension layer
mkdir -p /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/collector-config
cp config* /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/collector-config
cd /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build && zip -r collector-extension.zip collector-config extensions
  adding: collector-config/ (stored 0%)
  adding: collector-config/config.yaml (deflated 54%)
  adding: extensions/ (stored 0%)
  adding: extensions/collector (deflated 65%)
bash-4.2# ls -al
total 272
drwxr-xr-x  5 root root   4096 Mar  1 21:20 .
drwxr-xr-x 11 root root   4096 Mar  1 21:20 ..
drwxr-xr-x  4 root root   4096 Mar  1 21:20 build
-rw-r--r--  1 root root   2736 Mar  1 21:20 collector.go
-rw-r--r--  1 root root    251 Mar  1 21:20 config.yaml
drwxr-xr-x  2 root root   4096 Mar  1 21:20 extension
-rw-r--r--  1 root root   4964 Mar  1 21:20 go.mod
-rw-r--r--  1 root root 219786 Mar  1 21:20 go.sum
drwxr-xr-x  2 root root   4096 Mar  1 21:20 lambdacomponents
-rw-r--r--  1 root root   2358 Mar  1 21:20 main.go
-rw-r--r--  1 root root   1624 Mar  1 21:20 Makefile
-rw-r--r--  1 root root   1042 Mar  1 21:20 print.go
-rw-r--r--  1 root root   3222 Mar  1 21:20 README.md
-rw-r--r--  1 root root      7 Mar  1 21:20 VERSION
bash-4.2# popd
/var/task/aws-otel-lambda/python
bash-4.2# ls -al
total 24
drwxr-xr-x  5 root root 4096 Mar  1 21:20 .
drwxr-xr-x 12 root root 4096 Mar  1 21:20 ..
-rwxr-xr-x  1 root root  758 Mar  1 21:20 build.sh
drwxr-xr-x  3 root root 4096 Mar  1 21:20 integration-tests
drwxr-xr-x  3 root root 4096 Mar  1 21:20 sample-apps
drwxr-xr-x  2 root root 4096 Mar  1 21:20 scripts
bash-4.2# cd ..
bash-4.2# ls -al
total 100
drwxr-xr-x 12 root root  4096 Mar  1 21:20 .
drwxr-xr-x  1 root root  4096 Mar  1 21:20 ..
drwxr-xr-x  5 root root  4096 Mar  1 21:20 adot
-rw-r--r--  1 root root  7850 Mar  1 21:20 CHANGELOG.md
-rw-r--r--  1 root root   309 Mar  1 21:20 CODE_OF_CONDUCT.md
-rw-r--r--  1 root root  3160 Mar  1 21:20 CONTRIBUTING.md
drwxr-xr-x  4 root root  4096 Mar  1 21:20 dotnet
drwxr-xr-x  9 root root  4096 Mar  1 21:20 .git
-rw-r--r--  1 root root    85 Mar  1 21:20 .gitattributes
drwxr-xr-x  4 root root  4096 Mar  1 21:20 .github
-rw-r--r--  1 root root   127 Mar  1 21:20 .gitignore
-rw-r--r--  1 root root   130 Mar  1 21:20 .gitmodules
drwxr-xr-x  4 root root  4096 Mar  1 21:20 go
drwxr-xr-x  7 root root  4096 Mar  1 21:20 java
-rw-r--r--  1 root root 10142 Mar  1 21:20 LICENSE
drwxr-xr-x  6 root root  4096 Mar  1 21:20 nodejs
-rw-r--r--  1 root root    67 Mar  1 21:20 NOTICE
drwxr-xr-x 11 root root  4096 Mar  1 21:20 opentelemetry-lambda
-rwxr-xr-x  1 root root  1626 Mar  1 21:20 patch-upstream.sh
drwxr-xr-x  5 root root  4096 Mar  1 21:20 python
-rw-r--r--  1 root root  2467 Mar  1 21:20 README.md
drwxr-xr-x  3 root root  4096 Mar  1 21:20 sample-apps
bash-4.2#

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L9-L12

This looks like it builds the SDK layer, using Docker, with volume mapping to a local directory, then puts the resulting zip as build/layer.zip. Is the opentelemetry-lambda container published somewhere such that i could use ...

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L14-L16

What does it need from the sample apps to complete this build?

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L18-L22

Now it installs opentelemetry-sdk-extension-aws into build/python.

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L24-L27

That seems to unzip the layers already created, and combine them.

https://github.com/aws-observability/aws-otel-lambda/blob/33c85b38836b7d2715d087ed5c1af0324c74e422/python/build.sh#L29-L31

@NathanielRN explained this:

One last complexity is that we rename it and replace it with our own "otel-instrument" because we want to add AWS X-Ray Python configuration values

https://github.com/aws-observability/aws-otel-lambda/blob/main/python/build.sh#L33-L35

And this creates the final layer as an archive (which i won't need).

At what point is patch-upstream.sh invoked?

So after i get the build working (in an image), i just need to COPY FROM all the files in the opentelemetry-lambda/python/src/build dir into the /opt dir of my lambda image?

smcoll avatar Mar 07 '22 19:03 smcoll

+1, this would be very useful to have

dknz2008 avatar Apr 03 '22 23:04 dknz2008

Could we also have this added to the roadmap and some indication of timeline https://github.com/orgs/aws-observability/projects/4

dknz2008 avatar Apr 04 '22 01:04 dknz2008

@dknz2008 we will evaluate but can't commit to a prioritization yet.

alolita avatar Apr 04 '22 01:04 alolita

@alolita any update if possible?

dknz2008 avatar Jul 14 '22 02:07 dknz2008

I also want this even for Docker Node.js environment to install otel in fewer steps

tmokmss avatar Oct 15 '22 04:10 tmokmss

This would great, obviously for all language/environments.

@alolita In the shorter term, it might be helpful to provide some guidance on how one could achieve this themselves.

jfloodnet avatar Oct 19 '22 11:10 jfloodnet

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

github-actions[bot] avatar Jan 22 '23 20:01 github-actions[bot]

This issue was closed because it has been marked as stale for 30 days with no activity.

github-actions[bot] avatar Feb 26 '23 20:02 github-actions[bot]

Yes, if publishing images is not on the roadmap, at least a user guide would be great. Should i open a separate issue?

It's been a long time since i gave this a shot, but i shared my progress and requests for context at https://github.com/aws-observability/aws-otel-lambda/issues/195#issuecomment-1061037790 - i was unable to get further at the time.

EDIT: oh, it looks like someone was able to figure it out - i can start from https://github.com/aws-observability/aws-otel-lambda/issues/209#issuecomment-1028393564

smcoll avatar Apr 18 '23 23:04 smcoll

Having the layer packaged as a container would make building custom lambda runtimes much easier. Although there is a workaround (download and unzip the layer), that is rather cumbersome, while building a container image from scratch with the contents of the layers seems rather trivial. I would be happy to contribute a PR to build container images for the layer, if there is interest from the maintainers.

miguelaferreira avatar Jul 10 '23 18:07 miguelaferreira