getmoto
Mocking lambda using Python 3.12 Runtime fails: `manifest for lambci/lambda:python3.12 not found`
I'm trying to write tests for a lambda function using the python 3.12 runtime.
This is failing with:
error running docker: 404 Client Error for http+docker://localhost/v1.43/images/create?tag=python3.12&fromImage=lambci%2Flambda: Not Found ("manifest for lambci/lambda:python3.12 not found: manifest unknown: manifest unknown")
Looking over the code here:
https://github.com/getmoto/moto/blob/77fbf518edbd78a427823380cc530bda079799c1/moto/awslambda/models.py#L1016-L1028
it looks like moto first tries to find an appropriate docker image for the target runtime in https://hub.docker.com/r/mlupin/docker-lambda/tags and then when it can't find a python 3.12 image there it falls back to https://hub.docker.com/r/lambci/lambda/tags
and then fails with that error.
I accept this isn't fundamentally a bug in moto and you can boil it down to "someone needs to go work on https://github.com/mLupine/docker-lambda/issues/26 upstream"
The thing I find surprising about this is that moto relies on third-party docker images and not the lambda images that AWS publish themselves: https://gallery.ecr.aws/lambda/python
Looking back over some old issues/PRs e.g:
- https://github.com/getmoto/moto/pull/4889
- https://github.com/getmoto/moto/issues/5886
- https://github.com/getmoto/moto/pull/5895
I cant really see any discussion of this, but I assume there is a reason. Anyone able to shed any light on this?
It also doesn't seem possible to tell moto to use (for example) public.ecr.aws/lambda/python:3.12 with the MOTO_DOCKER_LAMBDA_IMAGE var due to the difference in naming conventions.
@bblommers if you're ok with this change then I can implement images from https://gallery.ecr.aws/lambda/, would make it the first repo to check.
Hi @chris48s!
I assume there is a reason
When we started using the lambCI images (I think back in 2017), the official AWS images weren't available yet. So we didn't really have a choice. :slightly_smiling_face:
Supporting the official repositories makes sense, although I'd be a little worried about making it the default option. Technically it should be interchangeably, but I don't have enough insight in the makeup of both Docker images to be confident that we can do that in a patch release and expect everything to work as before.
How about we just support MOTO_DOCKER_LAMBDA_IMAGE=public.ecr.aws/lambda/python:3.12? That's a one-line change, if I'm not mistaken:
- image_ref = f"{image_repo}:{self.run_time}"
+ image_ref = image_repo if ":" in image_repo else f"{image_repo}:{self.run_time}"
Hi thanks for the response.
I tried out applying the patch you suggested locally and ran my tests with MOTO_DOCKER_LAMBDA_IMAGE=public.ecr.aws/lambda/python:3.12 pytest and I got
botocore.exceptions.ClientError: An error occurred (502) when calling the Invoke operation (reached max retries: 4): <!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code: 502</p>
<p>Message: Bad Gateway.</p>
<p>Error code explanation: 502 - Invalid responses from another server/proxy.</p>
</body>
</html>
in site-packages/botocore/client.py
I also tried changing the runtime and invoking with MOTO_DOCKER_LAMBDA_IMAGE=public.ecr.aws/lambda/python:3.11 pytest and MOTO_DOCKER_LAMBDA_IMAGE=public.ecr.aws/lambda/python:3.10 pytest just in case it was something specific to that runtime, but I got the same result.
I guess there is some additional difference between the official and third party images here.
I haven't yet tried reducing this to a minimal repro independent of my application's test suite though.
Yup, mlupin's images have quite a bit of customisation:
- dedicated init process https://github.com/mLupine/docker-lambda/blob/master/runtimes/provided.al2-x86_64/run/init.go
- depending on runtime, it injects "default" lambda(https://github.com/mLupine/docker-lambda/blob/024d9875942d6ab8b7f539bf989023368b6b128a/dump-fs/dump-python38/handler.py#L18), this is done for every runtime supported by lambda
- The init process from 1 wraps lambda runtime emulator to produce cost/time/memory usage stats at the end, just like the real lambda
This is a full project itself! Sadly last commit to repo is in August last year and none of the forks have p3.12. So to keep existing functionality mLupine's repo should be forked, updated and maintained. Or similar level of customisation would have to be added to the public.ecr.aws images. Either way it's not a small job. Whaddaya think @bblommers ?
Thanks for the context @rafcio19. I assumed there were some differences, but I didn't know the extent of it.
There is https://github.com/shogo82148/docker-lambda, which is a fork of mlupine that does support python 3.12. It seems to have automated releases, actually, so that is already promising. The naming scheme is different though, so we'd still need the patch that I mentioned above.
There's also a financial argument here. It looks like shogo82418 is a good replacement for mlupine, but it is hosted on ECR, which tends to be expensive. If Moto makes that the default and they suddenly get a ton of additional outbound traffic, that increases the cost for them quite dramatically. That's quite unethical, and also increases the potential that they will disappear as an image provider.
So no easy solution, in short. We can do nothing and hope that mlupin comes back, we can take on the major job of maintaining our own, or we increase the (financial) burden on a third-party (and hope they stick around!).
A fourth option would be to take the base-image provided by AWS, and build the image that we need on-demand (at execution time). It looks like that's the way LocalStack does it.
https://github.com/localstack/localstack/blob/3a54f21ba4d4cfb75fea569fbf489d51ffca1b49/localstack-core/localstack/services/lambda_/invocation/docker_runtime_executor.py#L198
It would be the most future-proof way of doing this, but also the most time-consuming.
There's also a financial argument here. It looks like
shogo82418is a good replacement formlupine, but it is hosted on ECR, which tends to be expensive. If Moto makes that the default and they suddenly get a ton of additional outbound traffic, that increases the cost for them quite dramatically. That's quite unethical, and also increases the potential that they will disappear as an image provider.
I haven't tested those images, but looking over the repo the auto builds also push them to:
- GitHub Container Registry: https://github.com/shogo82148/docker-lambda/pkgs/container/lambda-python
- DockerHub: https://hub.docker.com/r/shogo82148/lambda-python/tags
so it seems like you could consume those images from other sources without costing them ECR bandwidth.
Just as a really quick test, I tried applying your patch locally again and ran
MOTO_DOCKER_LAMBDA_IMAGE=ghcr.io/shogo82148/lambda-python:build-3.12 pytest
That failed with
botocore.parsers.ResponseParserError: Unable to parse response (mismatched tag: line 6, column 6), invalid XML received. Further retries may succeed:
b'<!DOCTYPE HTML>\n<html lang="en">\n <head>\n <meta charset="utf-8">\n <title>Error response</title>\n </head>\n <body>\n <h1>Error response</h1>\n <p>Error code: 502</p>\n <p>Message: Bad Gateway.</p>\n <p>Error code explanation: 502 - Invalid responses from another server/proxy.</p>\n </body>\n</html>\n'
which is a similar (but not identical) result to what I saw with the official AWS image.
@chris48s the build image is used for building dependencies only, then compiled files are copied to the run image. With @bblommers's patch try MOTO_DOCKER_LAMBDA_IMAGE=shogo82148/lambda-python:3.12.2024.10.18 pytest, that works for me.
Right, yes. Thanks. Error between chair and keyboard there :facepalm:
It feels like whichever way you go on which images are used by default it would be useful to have that little patch allowing MOTO_DOCKER_LAMBDA_IMAGE to be a specific tagged image anyway as it provides users with more of an escape hatch.
@chris48s I just created a dev-release that contains the patch - please let us know if you run into any issues with that:
https://pypi.org/manage/project/moto/release/5.0.19.dev7/
Yep. 5.0.19.dev7 allows me to work around this with MOTO_DOCKER_LAMBDA_IMAGE. Thanks.
More generally, I think there is still some value in re-evaluating the defaults in a future release if the existing repos don't keep pace with new runtimes that AWS are adding.