Support local development on machines running Podman instead of Docker

[mario-d-s]

trafficstars

Describe your idea/feature/enhancement

I'm on Fedora 31 which, by default, ships with Podman instead of Docker. By installing podman-docker on top, most Docker workflows work pretty great out of the box. I've really bought into this idea mainly because Podman is lighter and doesn't require root priviliges.

This fails where tools depend on the Docker's "proprietary" protocol to manage containers, as is the case with SAM CLI.

Proposal

My knowledge about the container ecosystem, the OCI and where tools like Docker (vs. Podman) fit into that exactly is pretty limited. The question is, can tools like AWS SAM be made to work for end-users like me in an easy fashion where the answer is not to install Docker proper?

This could be a change to the SAM CLI so as not to have a hard dependency on there being a Docker socket. This could also be a change to Podman where they emulate the Docker API / socket. This could be a change to Python Docker SDK to work with both the Docker API and Podman's varlink-based API.

I'm just looking for a solution as an end user.

Things to consider:

1. Will this require any updates to the SAM Spec -> No

[luizcorreia]

Hello, this would be a very welcome feature, I have been using fedora since version 27 and the fact that the docker is not compatible with the cgroup bothers a lot. I'm not going to change the system for this, would there be any other way for sam to work local without the local docker? Is a version with podman possible?

[FallenWarrior2k]

Podman supports Docker's API, so you can use the podman-docker package, which symlinks /run/docker.sock to /run/podman/podman.sock and systemctl enable podman.socket to use applications depending on Docker's API with Podman. This also works with sam, although you have to manually create a default network using sudo podman network create default, as the default network is called podman and sam seems to depend on a default network existing.

[rkraneis]

Hmm, following your steps, @FallenWarrior2k, I still end up with Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running? :/

$ docker --version
podman version 3.0.1

$ sudo docker network ls
NAME     VERSION  PLUGINS
podman   0.4.0    bridge,portmap,firewall,tuning
default  0.4.0    bridge,portmap,firewall,tuning,dnsname

[FallenWarrior2k]

Are you running sam as root or as your regular user? You need to run it as root for this to work.

Also, after some further experimentation, I found that there's some issues still, such as "$num is not a valid stream" and some other problems that may or may not be spurious.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, March 12, 2021 3:54 PM, René Kraneis @.***> wrote:

Hmm, following your steps, @FallenWarrior2k, I still end up with Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running? :/

$ docker --version podman version 3.0.1

$ sudo docker network ls NAME VERSION PLUGINS podman 0.4.0 bridge,portmap,firewall,tuning default 0.4.0 bridge,portmap,firewall,tuning,dnsname

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[rkraneis]

Indeed, running as root works. I guess there also should be a way to make the socket available to select non-root users.

[FallenWarrior2k]

Do remember that comes with the same security implications that opening up your Docker socket comes with, i.e. trivial passwordless root escalation.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, March 12, 2021 4:52 PM, René Kraneis @.***> wrote:

Indeed, running as root works. I guess there also should be a way to make the socket available to select non-root users.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

Are you running sam as root or as your regular user? You need to run it as root for this to work. Also, after some further experimentation, I found that there's some issues still, such as "$num is not a valid stream" and some other problems that may or may not be spurious. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ … On Friday, March 12, 2021 3:54 PM, René Kraneis @.***> wrote: Hmm, following your steps, @FallenWarrior2k, I still end up with Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running? :/ $ docker --version podman version 3.0.1 $ sudo docker network ls NAME VERSION PLUGINS podman 0.4.0 bridge,portmap,firewall,tuning default 0.4.0 bridge,portmap,firewall,tuning,dnsname — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#1668 (comment)), or unsubscribe.

So my happy was short, using Docker API doesn't work. this was the trace:

2021-03-22 10:40:02,386 | Starting a timer for 100 seconds for function 'cenario03Function'
Exception in thread Thread-8:
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 672, in urlopen
  File "urllib3/connectionpool.py", line 421, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3/connectionpool.py", line 416, in _make_request
  File "http/client.py", line 1369, in getresponse
  File "http/client.py", line 310, in begin
  File "http/client.py", line 279, in _read_status
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 720, in urlopen
  File "urllib3/util/retry.py", line 400, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 672, in urlopen
  File "urllib3/connectionpool.py", line 421, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3/connectionpool.py", line 416, in _make_request
  File "http/client.py", line 1369, in getresponse
  File "http/client.py", line 310, in begin
  File "http/client.py", line 279, in _read_status
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "samcli/local/docker/container.py", line 301, in wait_for_logs
  File "docker/models/containers.py", line 96, in attach
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/container.py", line 63, in attach
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/client.py", line 447, in _check_is_tty
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/container.py", line 758, in inspect_container
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 230, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[ghost]

And some time I get this error:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "samcli/local/docker/container.py", line 303, in wait_for_logs
  File "samcli/local/docker/container.py", line 340, in _write_container_output
  File "docker/types/daemon.py", line 32, in __next__
  File "docker/api/client.py", line 406, in <genexpr>
  File "docker/utils/socket.py", line 169, in demux_adaptor
ValueError: 10 is not a valid stream

[ghost]

This was the trace from another lambda, change full path to '...' on this just to security The call: sudo sam local start-api --debug --log-file logs.txt --profile luiz --region sa-east-1 -t template.yml

podman-docker working and podman.socket enabled

$ docker --version
podman version 3.0.1

$ sudo docker network ls
NAME     VERSION  PLUGINS
podman   0.4.0    bridge,portmap,firewall,tuning
default  0.4.0    bridge,portmap,firewall,tuning,dnsname

2021-03-22 10:59:50,564 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7", "Cache-Control": "max-age=0", "Connection": "keep-alive", "Host": "127.0.0.1:3000", "Sec-Ch-Ua": "\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"", "Sec-Ch-Ua-Mobile": "?0", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36", "X-Forwarded-Port": "3000", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7"], "Cache-Control": ["max-age=0"], "Connection": ["keep-alive"], "Host": ["127.0.0.1:3000"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"], "X-Forwarded-Port": ["3000"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/estados", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "127.0.0.1:3000", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "127.0.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/estados", "protocol": "HTTP/1.1", "requestId": "9b96f1ba-bd3e-4a53-b917-516c96a79f01", "requestTime": "22/Mar/2021:13:57:54 +0000", "requestTimeEpoch": 1616421474, "resourceId": "123456", "resourcePath": "/estados", "stage": "dev"}, "resource": "/estados", "stageVariables": null, "version": "1.0"}
2021-03-22 10:59:50,564 | Found one Lambda function with name 'Cenario02Function'
2021-03-22 10:59:50,564 | Invoking index.handler (nodejs12.x)
2021-03-22 10:59:50,564 | Environment variables overrides data is standard format
2021-03-22 10:59:50,564 | Resolving code path. Cwd=/.../cenario02, CodeUri=/.../cenario02
2021-03-22 10:59:50,564 | Resolved absolute path to code is /.../cenario02
2021-03-22 10:59:50,565 | Code /.../cenario02 is not a zip/jar file
2021-03-22 10:59:50,565 | arn:aws:lambda:sa-east-1:369588294699:layer:smiles-poc-arq-propositiva-layer:5 is already cached. Skipping download
2021-03-22 10:59:50,585 | Skip pulling image and use local one: samcli/lambda:nodejs12.x-d6091ec43c28f36beae7336ee.

2021-03-22 10:59:50,585 | Mounting /.../cenario02 as /var/task:ro,delegated inside runtime container
2021-03-22 10:59:50,923 | Starting a timer for 30 seconds for function 'Cenario02Function'
Exception in thread Thread-14:
Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "samcli/local/docker/container.py", line 303, in wait_for_logs
  File "samcli/local/docker/container.py", line 340, in _write_container_output
  File "docker/types/daemon.py", line 32, in __next__
  File "docker/api/client.py", line 406, in <genexpr>
  File "docker/utils/socket.py", line 169, in demux_adaptor
ValueError: 10 is not a valid stream

2021-03-22 10:59:51,388 | Cleaning all decompressed code dirs
2021-03-22 10:59:51,388 | Invalid lambda response received: Lambda response must be valid json
2021-03-22 10:59:51 127.0.0.1 - - [22/Mar/2021 10:59:51] "GET /estados HTTP/1.1" 502 -
2021-03-22 10:59:51 127.0.0.1 - - [22/Mar/2021 10:59:51] "GET /favicon.ico HTTP/1.1" 403 -

[meeuw]

I have something working with:

podman system service --time=0 unix:///tmp/podman.sock &
DOCKER_HOST=unix:///tmp/podman.sock sam build --use-container

I also need to set SELinux in Permissive mode (setenforce Permissive) for this to work.

edit:

It's also possible to start the podman service using systemctl --user start podman, DOCKER_HOST should then be set to /run/user/$UID/podman/podman.sock

[mario-d-s]

Fedora Magazine recently published an article that seems very interesting and related to this. Starting from Podman version 3.0, it has improved support for emulating the Docker socket, and it even supports docker-compose. It's the same idea as the one from @meeuw above.

Specifically, it appears you can do (at least on Fedora 34):

systemctl --user enable podman.socket
systemctl --user start podman.socket
systemctl --user status podman.socket
export DOCKER_HOST=///run/user/$UID/podman/podman.sock

Haven't tested if this makes sam local work yet but it looks promising.

[meeuw]

Not sure if it would help but I was thinking of writing a cloudformation template which deploys an EC2 instance with Fedora and SELinux enabled so the aws-sam-cli devs can easily maintain this.

[copperii]

Export format should be: systemctl --user enable podman.socket systemctl --user start podman.socket systemctl --user status podman.socket export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

Without "unix:" part fails with "docker.errors.DockerException: Invalid bind address format:"

[athewsey]

IMO the Docker Desktop licensing changes of August this year (per docker.com, theregister) make this request even more relevant: Not just for Linux desktop users but anybody else that may now have additional reasons to want non-Docker-based container tools...

[shearn89]

Just bumping in to this myself, I'm on an M1 Mac and happy to test anything that needs testing!

[shearn89]

This still appears to be an issue on Apple Silicon.

I've done the following, based on various bits of internet research:

1. Expose podman machine socket: podman machine ssh -L '/tmp/podman.sock:/run/user/1000/podman/podman.sock' -N &
2. Set socket for sam: export DOCKER_HOST=unix:///tmp/podman.sock

This gets me to the stage where I can run sam local invoke, but it errors:

Invoking hello-world (go1.x)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-go1.x
Building image....................
Failed to build Docker Image
NoneType: None
Error: Error building docker image: error building at STEP "RUN mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie": error while running runtime: exit status 1

I'm using the simple 'hello world' example from sam init, in Go, with Zip packaging.

I tried the command from here, but same result...

Turns out I think that link is for running arm containers on x86. I'm investigating some other resources to see if I can get this working on my Mac.

[shearn89]

Turns out I had forgotten to change the arch in the SAM template. On further investigation there is also no support for arm64 architecture for the Go runtime, instead requiring a custom runtime.

That got me a bit further, until I bumped into this:

$> sam local invoke
...
docker.errors.APIError: 500 Server Error: Internal Server Error ("make cli opts(): error making volume mountpoint for volume /Users/ajshearn/repos/sam-go-ping/.aws-sam/build/HelloWorldFunction: mkdir /Users: operation not permitted")

Generally, this is a terrible user experience. I knew Apple moving to arm64 would be painful, but this is just painful! On the plus side, it's probably pushing me to do more stuff via CI/CD than via local testing. But it would be nice to be able to test locally using native tooling...

[shymega]

I can confirm setting DOCKER_HOST works for building, but for me on my F35 system, it does not for deployment. I haven't tested sam local yet.

I get this error:

Error: Unable to upload artifact awsgamingtelegrambotfunction:python3.8-v1 referenced by ImageUri parameter of AwsGamingTelegramBotFunction resource.
UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

I suspect sam deploy needs to pay attention to the $DOCKER_HOST environmental variable as well - maybe?

[ananthb]

@shymega running the podman machine ssh -L ... command to expose the socket fails on my M1 mac now. Podman machine says -L is an unknown flag. Any idea what has changed?

[shymega]

I don't think you meant to tag me - I only reported sam deploy not working.

On Tue, 2 Aug 2022, at 4:18 PM, Ananth Bhaskararaman wrote:

@shymega https://github.com/shymega running the podman machine ssh -L ... command to expose the socket fails on my M1 mac now. Podman machine says -L is an unknown flag. Any idea what has changed?

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-sam-cli/issues/1668#issuecomment-1202801587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKF2QGSPEEBUV7NUESWEOTVXE34ZANCNFSM4J3LZ2VQ. You are receiving this because you were mentioned.Message ID: @.***>

-- Kind regards,

Dom Rodriguez (also known as shymega)

[ananthb]

You're absolutely right!

@shearn89 i was hoping you could chime in on how you got sam working with podman on the M1.

[shearn89]

I don't think I did, I think I double checked with work and just used Docker instead... 😢

[mindacrobatic]

I would like to refresh this topic a bit since I still think that a lightweighted container engine like Podman should be supported by AWS SAM.

What is working right now and what is not? Let me show you what I am doing on a system with the following versions: Fedora version 36 AWS CLI: 2.7.29 AWS SAM CLI: 1.56.0

# I enable and start the podman socket
systemctl --user enable podman.socket
systemctl --user start podman.socket
systemctl --user status podman.socket

# I link DOCKER_HOST to the podman socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

# I create a basic example (python based with ZIP and not image version)
sam init

# I initialized the sam app in the folder sam-app
cd sam-app

# I try to build it using a container
sam build --use-container

This fails with the following error message:

Traceback (most recent call last):
  File "samcli/local/docker/manager.py", line 85, in create
  File "samcli/local/docker/manager.py", line 163, in pull_image
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("error normalizing image: normalizing name for compat API: invalid reference format")

The above exception was the direct cause of the following exception:
raceback (most recent call last):
  File "samcli/__main__.py", line 12, in <module>
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "click/decorators.py", line 73, in new_func
  File "click/core.py", line 610, in invoke
  File "samcli/lib/telemetry/metric.py", line 181, in wrapped
  File "samcli/lib/telemetry/metric.py", line 129, in wrapped
  File "samcli/lib/utils/version_checker.py", line 41, in wrapped
  File "samcli/cli/main.py", line 87, in wrapper
  File "samcli/commands/build/command.py", line 201, in cli
  File "samcli/commands/build/command.py", line 262, in do_cli
  File "samcli/commands/build/build_context.py", line 252, in run
  File "samcli/lib/build/app_builder.py", line 221, in build
  File "samcli/lib/build/build_strategy.py", line 80, in build
  File "samcli/lib/build/build_strategy.py", line 90, in _build_functions
  File "samcli/lib/build/build_strategy.py", line 174, in build_single_function_definition
  File "samcli/lib/build/app_builder.py", line 654, in _build_function
  File "samcli/lib/build/app_builder.py", line 819, in _build_function_on_container
  File "samcli/local/docker/manager.py", line 115, in run
  File "samcli/local/docker/manager.py", line 90, in create
samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it from docker.

I advanced a bit further by pulling the image myself and then rerunning the command again:

docker pull public.ecr.aws/sam/build-python3.9:latest-x86_64
sam build --use-container

It starts to build using the downloaded container, but then fails:

Running PythonPipBuilder:ResolveDependenciesTraceback (most recent call last):
  File "samcli/__main__.py", line 12, in <module>
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "click/decorators.py", line 73, in new_func
  File "click/core.py", line 610, in invoke
  File "samcli/lib/telemetry/metric.py", line 181, in wrapped
  File "samcli/lib/telemetry/metric.py", line 129, in wrapped
  File "samcli/lib/utils/version_checker.py", line 41, in wrapped
  File "samcli/cli/main.py", line 87, in wrapper
  File "samcli/commands/build/command.py", line 201, in cli
  File "samcli/commands/build/command.py", line 262, in do_cli
  File "samcli/commands/build/build_context.py", line 252, in run
  File "samcli/lib/build/app_builder.py", line 221, in build
  File "samcli/lib/build/build_strategy.py", line 80, in build
  File "samcli/lib/build/build_strategy.py", line 90, in _build_functions
  File "samcli/lib/build/build_strategy.py", line 174, in build_single_function_definition
  File "samcli/lib/build/app_builder.py", line 654, in _build_function
  File "samcli/lib/build/app_builder.py", line 831, in _build_function_on_container
  File "samcli/local/docker/container.py", line 335, in wait_for_logs
  File "samcli/local/docker/container.py", line 400, in _write_container_output
  File "docker/types/daemon.py", line 32, in __next__
  File "docker/api/client.py", line 406, in <genexpr>
  File "docker/utils/socket.py", line 169, in demux_adaptor
ValueError: 45 is not a valid stream

Using --debug flag did not yield to (for me) usable information. Also, running sam with sudo permissions does not help for me.

Appendix: For some reasons, after some playing around with it, it was suddenly building a moment ago. But I cannot reproduce what I did. Also, I remember that some weeks ago, I was able to run build in a similar way (I think I was also playing around with sudo setenforce 0), but then sam deploy failed.

Also from my side, it would be more than welcomed to be able to use a great tool like Podman with AWS SAM.

[lvthillo]

on my mac (x68):

• Installed podman w brew:
• export DOCKER_HOST='unix:///Users/<my-user>/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Now aws-sam-cli works well with:

• sam local invoke
• sam local start-api

But this still fails: sam build --use-container

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/create?tag=latest-x86_64&fromImage=public.ecr.aws%2Fsam%2Fbuild-python3.9%3Alatest-x86_64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 160, in pull_image
    result_itr = self.docker_client.api.pull(image_name, tag=tag, stream=True, decode=True)
  File "/usr/local/lib/python3.10/site-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("normalizing image: normalizing name for compat API: invalid reference format")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 85, in create
    self.pull_image(image_name)
  File "/usr/local/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 163, in pull_image
    raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("normalizing image: normalizing name for compat API: invalid reference format")

The above exception was the direct cause of the following exception:
...
 Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it from docker

It works when I manually pull the image first.

[jfuss]

We currently only support tools that are compatible with docker, as that is the underlying tech we built on (through Docker support). At this time, we are not planning to directly support Podman but from comments above seems like there are workarounds the community has to enable this directly.

Closing.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rkujawa]

@jfuss Please reconsider this decision on a business level. AWS already has migrated away from Docker for the purpose of running Kubernetes (EKS). Docker is known to be hostile towards both open source community and companies that do not pay for the services they provide, that were previously provided for free. This makes Docker unattractive and a lot of companies are looking for alternatives to Docker.

As an AWS customer I don't feel the need to additionally pay Docker to use official AWS tools. Creating such dependency is harmful for your business.

[dmarinuswoodwing]

Not sure how you're using podman but you can also use docker for free (using moby-engine).

[com4]

here's a quick attempt at fixing the issues i've run across:

• sam build -u gave the error Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it [...]. This is because the podman engine is less forgiving with a tag appended to an image name. the fix is to fully split the image and tag

diff --git a/samcli/local/docker/manager.py b/samcli/local/docker/manager.py
index 5780852d..7835cca0 100644
--- a/samcli/local/docker/manager.py
+++ b/samcli/local/docker/manager.py
@@ -142,7 +142,12 @@ class ContainerManager:
             If the Docker image was not available in the server
         """
         if tag is None:
-            tag = image_name.split(":")[1] if ":" in image_name else "latest"
+            _image_name_split = image_name.split(":")
+            # Separate the image_name from the tag so less forgiving docker clones
+            # (podman) get the image name as the URL they expect. Official docker seems
+            # to clean this up internally.
+            tag = _image_name_split[1] if ":" in image_name else "latest"
+            image_name = _image_name_split[0]
         # use a global lock to get the image lock
         with self._lock:
             image_lock = self._lock_per_image.get(image_name)

• Next sam local start-api wasn't working. this is because sam is checking that the base image is the most recent version on each invoke but fails because podman is missing the distribution api (https://github.com/containers/podman/issues/17726). the fix is to favor a stale image over crashing.

diff --git a/samcli/local/docker/lambda_image.py b/samcli/local/docker/lambda_image.py
index f4b3a261..314c6aaf 100644
--- a/samcli/local/docker/lambda_image.py
+++ b/samcli/local/docker/lambda_image.py
@@ -12,6 +12,7 @@ from pathlib import Path
 from typing import Optional
 
 import docker
+from docker.errors import NotFound as DockerNotFound
 
 from samcli.commands.local.cli_common.user_exceptions import ImageBuildException
 from samcli.commands.local.lib.exceptions import InvalidIntermediateImageError
@@ -447,7 +448,22 @@ class LambdaImage:
         if self.skip_pull_image or self.force_image_build:
             return
 
-        if self.is_base_image_current(image_name):
+        # if we can't check to see if the image is up-to-date, warn but don't crash.
+        try:
+            base_image_is_current = self.is_base_image_current(image_name)
+        except DockerNotFound as e:
+            # Maybe podman.. assume it's the latest because we can't check
+            # https://github.com/containers/podman/issues/17726
+            LOG.warn(
+                "Cannot check if base image is current because an error was returned from "
+                "the Docker daemon API. This might be a Docker engine clone. Proceeding with "
+                "possibly stale image."
+            )
+            LOG.debug("Error response from Docker Engine", exc_info=e)
+            self.skip_pull_image = True
+            return
+
+        if base_image_is_current:
             self.skip_pull_image = True
             LOG.info("Local image is up-to-date")
         else:

[ananthb]

@com4 do you know which docker API is different in Podman? That can be reported and fixed upstream too.