Failure with ARM container running in AWS CodeBuild

[macbutch]

trafficstars

This same setup works fine on my MacBook Pro - same Dockerfile and build, same docker-slim invocation. I'm trying to get the same setup to work in CodeBuild. It may be worth mentioning - could be related - that the image I'm building is for armhf but I'm running on x86_64 on the host.

Here's the output when I run in CodeBuild:

[Container] 2019/12/12 06:34:34 Running command /tmp/docker-slim build --http-probe=false --continue-after 20 mosquitto 
docker-slim[build]: state=started 
docker-slim[build]: info=params target=mosquitto continue.mode=timeout 
docker-slim[build]: state=image.inspection.start 
docker-slim[build]: info=image id=sha256:91b3503e148d20dc7fb30893771a0d653498f4d3efc53d066447c22a38207ecc size.bytes=178925852 size.human=179 MB 
docker-slim[build]: info=image.stack index=0 name='balenalib/armv7hf-alpine:3.9-run' id='sha256:ab2e215257f0e5ad3a590e54ab721b7e4b6195b52dad28a21fb5ad65450145c7' 
docker-slim[build]: info=image.stack index=1 name='mosquitto:latest' id='sha256:91b3503e148d20dc7fb30893771a0d653498f4d3efc53d066447c22a38207ecc' 
docker-slim[build]: info=image.exposed_ports list='1883,8883' 
docker-slim[build]: state=image.inspection.done 
docker-slim[build]: state=container.inspection.start 
docker-slim[build]: info=container status=created name=dockerslimk_2107_20191212063435 id=49e3da49e38b0de1212657b2d33a5b3dd5e03114236f4b7917a0013e3d4d29a1 
docker-slim[build]: info=cmd.startmonitor status=sent 
docker-slim[build]: info=event.error status=received data=SensorError{Op:sensor.ptrace.Run/target.Start,Kind:call.error,Wrapped:{Type=*os.PathError,Info=fork/exec /usr/bin/entry.sh: exec format error,Line:88,File:github.com/docker-slim/docker-slim@/internal/app/sensor/monitors/ptrace/monitor.go}} 
docker-slim[build]: state=exited version=linux|Transformer|1.26.1|2ec04e169b12a87c5286aa09ef44eac1cea2c7a1|2019-11-28_04:37:59PM 
 
[Container] 2019/12/12 06:34:35 Command did not exit successfully /tmp/docker-slim build --http-probe=false --continue-after 20 mosquitto exit status 132 
[Container] 2019/12/12 06:34:35 Phase complete: BUILD State: FAILED 
[Container] 2019/12/12 06:34:35 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: /tmp/docker-slim build --http-probe=false --continue-after 20 mosquitto. Reason: exit status 132 

I assume this is the relevant part:

{Type=*os.PathError,Info=fork/exec /usr/bin/entry.sh: exec format error,Line:88,File:github.com/docker-slim/docker-slim@/internal/app/sensor/monitors/ptrace/monitor.go}

On my MacBook I get this:

$ .../docker-slim build --http-probe=false --continue-after 20 mosquitto
docker-slim[build]: state=started
docker-slim[build]: info=params target=mosquitto continue.mode=timeout
docker-slim[build]: state=image.inspection.start
docker-slim[build]: info=image id=sha256:6c325f64bc1714ff4317ea1e67e6abf39631b6ace1acf35917ec608a07194b28 size.bytes=175693956 size.human=176 MB
docker-slim[build]: info=image.stack index=0 name='balenalib/armv7hf-alpine:3.9-run' id='sha256:ab2e215257f0e5ad3a590e54ab721b7e4b6195b52dad28a21fb5ad65450145c7'
docker-slim[build]: info=image.stack index=1 name='mosquitto:latest' id='sha256:6c325f64bc1714ff4317ea1e67e6abf39631b6ace1acf35917ec608a07194b28'
docker-slim[build]: info=image.exposed_ports list='1883,8883'
docker-slim[build]: state=image.inspection.done
docker-slim[build]: state=container.inspection.start
docker-slim[build]: info=container status=created name=dockerslimk_74755_20191212215300 id=446d664664d0a424fb7f8fef507771448feed30226eb4a93b09c80fc74c037b6
docker-slim[build]: info=cmd.startmonitor status=sent
docker-slim[build]: info=event.startmonitor.done status=received
docker-slim[build]: info=container name=dockerslimk_74755_20191212215300 id=446d664664d0a424fb7f8fef507771448feed30226eb4a93b09c80fc74c037b6 target.port.list=[32771,32770] target.port.info=[1883/tcp => 0.0.0.0:32771,8883/tcp => 0.0.0.0:32770] message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER'
docker-slim[build]: info=continue.after mode=timeout message='no input required, execution will resume after the timeout'
docker-slim[build]: info=prompt message='waiting for the target container (20 seconds)'
docker-slim[build]: info=event message='done waiting for the target container'docker-slim[build]: state=container.inspection.finishing
docker-slim[build]: state=container.inspection.artifact.processing
docker-slim[build]: state=container.inspection.done
docker-slim[build]: state=building message='building minified image'
docker-slim[build]: state=completed
docker-slim[build]: info=results status='MINIFIED BY 37.37X [175693956 (176 MB) => 4701356 (4.7 MB)]'
docker-slim[build]: info=results  image.name=mosquitto.slim image.size='4.7 MB' data=true
docker-slim[build]: info=results  artifacts.location='/Users/mark/Downloads/dist_mac/.docker-slim-state/images/6c325f64bc1714ff4317ea1e67e6abf39631b6ace1acf35917ec608a07194b28/artifacts'
docker-slim[build]: info=results  artifacts.report=creport.json
docker-slim[build]: info=results  artifacts.dockerfile.original=Dockerfile.fat
docker-slim[build]: info=results  artifacts.dockerfile.new=Dockerfile
docker-slim[build]: info=results  artifacts.seccomp=mosquitto-seccomp.json
docker-slim[build]: info=results  artifacts.apparmor=mosquitto-apparmor-profile
docker-slim[build]: state=done
docker-slim[build]: info=report file='slim.report.json'

I'm going to keep hacking away at it for now but any suggestions would be very welcome.

[brainstorm]

In regular docker you'd need to go for buildx when going for cross-arch builds instead of build, IIRC. Not sure if docker-slim implements that though.

[macbutch]

Thanks @brainstorm, I'll give that a try too but just wanted to clarify that I'm not using buildx on my locals builds and there's no issue. This issue seems to only kick in in AWS CodeBuild (meaning docker-in-docker).

[brainstorm]

Ah, I see... when I saw exec format error I thought that this is a typical "running x86 binary on ARM or viceversa", but docker in docker is another story/beast. Could you point to the buildspec.yml? This one is not using ARM machines, but that's what I'm using ATM, can you share yours?:

https://github.com/umccr/umccrise/blob/master/buildspec.yml

I didn't know CodeBuild supported ARM instances, interesting.

[macbutch]

Hey, that looks super helpful. I'm not doing the docker-in-docker magic you're doing so that might be part of it, let me hack mine to be closer to yours and see if it does anything.

There are a couple of other things...

I'm not sure when they turned them but, yes, AWS provides ARM build containers now in CodeBuild. They're billed at $0.015/minute for a large vs $0.02 for the x86 large which is cool. I'm not using those (but I did also try that).

What I'm doing is building in CodeBuild on a x86 host but using the balena images which have qemu configured inside the container to let you run ARM stuff in a container on x86. You can read about the magic here: https://www.balena.io/docs/reference/base-images/base-images/#building-arm-containers-on-x86-machines

when I saw exec format error I thought that this is a typical "running x86 binary on ARM or viceversa"

Honestly, you're probably right. My suspicion is that I need some way to wrap the sensor commands so they run using qemu but I'm out of my depth honestly.

[brainstorm]

Feel free to share the buildspec over here and I can take a quick look. I'm familiar with the qemu stuff and cross compilation you mention (from my https://github.com/dockcross experimentation and work).