projectatomic
atomicapp fails due to 'docker images' returning '400 Bad Request'
When atomicapp inside an existing container image tries to run 'docker images' it fails with the following:
2016-03-10 09:15:47,702 - atomicapp.install - INFO - App name is submod/helloapache:0.1.11, will be populated to /atomicapp
2016-03-10 09:15:47,702 - atomicapp.utils - INFO - atomicapp.status.info.message=Loading app submod/helloapache:0.1.11 .
2016-03-10 09:15:47,703 - atomicapp.utils - INFO - atomicapp.status.answer.message={"general": {"namespace": "default", "provider": "kubernetes"}}
2016-03-10 09:15:47,703 - atomicapp.nulecule_base - DEBUG - {'namespace': 'default', 'provider': 'kubernetes'}
Error response from daemon: 400 Bad Request: malformed Host header
Traceback (most recent call last):
File "/usr/bin/atomicapp", line 9, in <module>
load_entry_point('atomicapp==0.1.11', 'console_scripts', 'atomicapp')()
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/cli/main.py", line 238, in main
cli.run()
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/cli/main.py", line 214, in run
args.func(args)
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/cli/main.py", line 51, in cli_run
ae = Run(**vars(args))
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/run.py", line 90, in __init__
install.install()
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/install.py", line 125, in install
self.nulecule_base.pullApp()
File "/usr/lib/python2.7/site-packages/atomicapp-0.1.11-py2.7.egg/atomicapp/nulecule_base.py", line 357, in pullApp
image_id = subprocess.check_output(check_cmd)
File "/usr/lib64/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['docker', 'images', '-q', 'submod/helloapache:0.1.11']' returned non-zero exit status 1
This is because docker on the host machine is docker-1.10.2-10.gitddbb15a.fc24.x86_64 and docker in the atomicapp container is docker-1.8.2-1.gitf1db8f2.fc22.x86_64.
The docker client in the atomicapp container is incompatible with the docker daemon running on the host.
To reproduce this:
- Install Fedora 24
yum install atomic dockeratomic run submod/helloapache:0.1.11
Running this inside the atomicapp container shows the issue:
[root@localhost atomicapp]# docker images
Error response from daemon: 400 Bad Request: malformed Host header
This is a result of docker daemon being built with golang 1.6 on Fedora 24:
https://github.com/golang/go/commit/6e11f45ebdbc7b0ee1367c80ea0a0c0ec52d6db5
https://groups.google.com/forum/#!topic/golang-codereviews/CPeFXzMJRzI
@stefwalter
I am unable to replicate this issue using projectatomic/helloapache against my host which is:
Client:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 22:37:33 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 22:37:33 2016
OS/Arch: linux/amd64
Running on Debian 8.1
Can I get the source / location of your example (submod/helloapache)?
I'm an idiot. You said Go 1.6... I've been downloading the official versions from the Docker site and I see that I'm using Go 1.5.3 haha.
Time to compile it! :D I'll get back to you on that.
@cdrage - another option may be to try to run the F24 rpm on F23. It might work since all deps should be baked in.
@stefwalter - what atomicapp container is submod/helloapache:0.1.11 based on? It looks like under the covers it is based on f22. Can you point to the Dockerfile?
@stefwalter - what atomicapp container is submod/helloapache:0.1.11 based on? It looks like under the covers it is based on f22. Can you point to the Dockerfile?
@dustymabe looks to be def related to 1.10.2 built on Go 1.6
The original submod/helloapache:0.1.11 (despite it being built via atomicapp 0.1.11) works fine on Docker 10.1.2 built on Go 1.5.3. I'm going to try and replacte 1.6.0 on my machine
I'm able to use docker (built with 1.6) completely fine with atomicapp 0.4.3. Is this a fedora-related issue?
bundles/latest/binary master ✔ 1d ⍉
▶ sudo atomicapp run projectatomic/helloapache --provider=docker
[INFO] - main.py - Action/Mode Selected is: run
[INFO] - base.py - Unpacking image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-b84624c673ac
[INFO] - container.py - Skipping pulling Docker image: projectatomic/helloapache
[INFO] - container.py - Extracting nulecule data from image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-b84624c673ac
587a8e403ff1653d42e0690eae841c5012088b663818a50ceab07aab045f6417
[INFO] - base.py - Provider not specified, using default provider - kubernetes
[WARNING] - plugin.py - Configuration option 'providerconfig' not found
[WARNING] - plugin.py - Configuration option 'providerconfig' not found
[INFO] - docker.py - Deploying to provider: Docker
9c36d3d5399ad2081e95c48ffa4361668b4975f8f6c23818373027f438a6b764
Your application resides in /var/lib/atomicapp/projectatomic-helloapache-b84624c673ac
Please use this directory for managing your application
bundles/latest/binary master ✔ 1d ⍉
▶ sudo atomicapp fetch projectatomic/helloapache
[INFO] - main.py - Action/Mode Selected is: fetch
[INFO] - base.py - Unpacking image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-216eb21bd98c
[INFO] - container.py - Skipping pulling Docker image: projectatomic/helloapache
[INFO] - container.py - Extracting nulecule data from image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-216eb21bd98c
45f9f729aa843764c7d6e8059d5c271467330d02d11b163118132985a5e38013
[INFO] - base.py - Provider not specified, using default provider - kubernetes
Your application resides in /var/lib/atomicapp/projectatomic-helloapache-216eb21bd98c
Please use this directory for managing your application
bundles/latest/binary master ✔ 1d
▶ docker version
Client:
Version: 1.11.0-dev
API version: 1.23
Go version: go1.6
Git commit: 9ff7439
Built: Fri Mar 11 16:56:14 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.0-dev
API version: 1.23
Go version: go1.6
Git commit: 9ff7439
Built: Fri Mar 11 16:56:14 2016
OS/Arch: linux/amd64
You need to use an old build of docker inside of the atomicapp image, built with golang 1.5.
The basic point of this bug is that atomicapp images can contain old versions of docker clients that are incompatible with future Atomic host versions.
@stefwalter have you tried using atomicapp:0.4.3 or use the projectatomic/helloapache example? The one within your example uses atomicapp 0.1.11
It's most likely just the old image that's being used.. We can direct users to use more up-to-date images in the FAQ if they experience this bug.
It's most likely just the old image that's being used.. We can direct users to use more up-to-date images in the FAQ if they experience this bug.
The point is that a packaging format like atomicapp shouldn't try to use its own copy of the docker client command to communicate with an arbitrary docker daemon (possibly older, possibly newer). This is a poor design that will break time and time again as docker changes, deprecates, and adds API.
Examples of how to fix this issue so it doesn't happen again: The atomicapp could use the docker client command from the host (perhaps copying it, or perhaps using it directly via a mount) or implement its own extra-compatible-in-perpetuity-docker-tool.
The point is that a packaging format like atomicapp shouldn't try to use its own copy of the docker client command to communicate with an arbitrary docker daemon (possibly older, possibly newer). This is a poor design that will break time and time again as docker changes, deprecates, and adds API.
Yeah, we are aware. This problem has come up in the past: https://github.com/projectatomic/atomicapp/issues/174
Examples of how to fix this issue so it doesn't happen again: The atomicapp could use the docker client command from the host (perhaps copying it, or perhaps using it directly via a mount) or implement its own extra-compatible-in-perpetuity-docker-tool.
So I have investigated before trying to use docker from the host. As you can see in the comment I hit some issues so I didn't try to go too much farther. The other thing I considered doing was chrooting back into the host to run the docker command so that we don't have the library issues, but I didn't know if that was just getting too complicated. We elected at the time to just error out with a good error message but you didn't see that message because of different issues (the golang 1.6 thing).
The other thing we have been wanting to do is use the docker-py library to communicate with the docker daemon rather than using a docker client at all. I guess that could possible have this same problem, but then again you could also make the same argument for using the docker command line (the command line options could change too).
I'm not sure of the best way to move forward right now
May be we should just ship Atomic App metadata and run it directly on the host using atomicapp, using the host's native docker client, compatible with the docker engine.
