Create Environment fails using Docker Toolbox 1.12

Open teamschwarz opened this issue 7 years ago • 9 comments

Error: Message is below. Running eval $(docker-machine env default) doesn't help :::::::::::::::::::: TLS configuration is invalid - make sure your DOCKER_TLS_VERIFY and DOCKER_CERT_PATH are set correctly. You might need to run eval "$(docker-machine env default)" Build step 'Execute shell' marked build as failure [ssh-agent] Stopped. Finished: FAILURE

Aug 15 '16 13:08 teamschwarz

Hi @teamschwarz, could you please provide more details, about how exactly (step-by-step) you have created ADOP stack, hard to say without details. OS and Docker version also would be useful. Thank you.

Aug 15 '16 16:08 anton-kasperovich

Windows 10. I've been using ADOP heavily on Docker Toolbox 1.11.x for months. The problem started after upgrading to 1.12. I run quickstart (git clone of adop-docker-compose a few days ago) to bring up the environment. i create a workspace/project and then load the Java cartridge (or use the one inside the Example workspace). Create environment always fails with the text above. Also, i think part of the same problem is trying to run the Chef Cartridge. the second step of the Chef pipeline: Sanity_Test -- error is: Could not read CA certificate "/root/.docker/ca.pem": open /root/.docker/ca.pem: no such file or directory This too ran fine before 1.12

Aug 16 '16 00:08 teamschwarz

I can confirm this issue, the same for me on Docker Version 1.12.0-a (build: 11213) . The strange thing is that because of broken TLS verification between local client and docker-machine, strange because i just for example added sleep statement in "cmd/compose" file:

@teamschwarz could you please check this workaround, In this case it should work, add sleep here https://github.com/Accenture/adop-docker-compose/blob/master/cmd/compose#L230 Like:

cp ${HOME}/.docker/machine/certs/ca.pem ${TEMP_CERT_PATH}/ca.pem
sleep 20
docker --tlsverify --tlscacert=${HOME}/.docker/machine/certs/ca.pem --tlscert=${TEMP_CERT_PATH}/cert.pem --tlskey=${TEMP_CERT_PATH}/key.pem -H=${DOCKER_HOST} version

And in root folder of docker-compose repository, please run this command:

./adop compose gen-certs //root/.docker

The output should be something like:

/Users/iniweb/docker_certs/cert.pem was generated successfully...
/Users/iniweb/docker_certs/ca.pem was generated successfully...
/Users/iniweb/docker_certs/key.pem was generated successfully...
Uploading certificates to Jenkins Slave at: //root/.docker

If so, you can re-run "Create_Environment" job or any other, your stack should work just as expected.

@nickdgriffin @SachinKSingh28: By default without sleep workaround, it's throw this error:

docker --tlsverify --tlscacert=/Users/iniweb/.docker/machine/certs/ca.pem --tlscert=/Users/iniweb/docker_certs/cert.pem --tlskey=/Users/iniweb/docker_certs/key.pem -H=tcp://53.90.82.17:2376 version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 21:15:28 2016
 OS/Arch:      darwin/amd64
The server probably has client authentication (--tlsverify) enabled. Please check your TLS client certification settings: Get https://53.90.82.17:2376/v1.24/version: remote error: bad certificate

Aug 21 '16 08:08 anton-kasperovich

i tried what you suggested but it did not work.

edited the compose file to add the sleep
ran the gen-certs command
tried create environment -- same error
restarted my DM and tried again -- same error
blew away my DM
ran quickstart to create and pull everything new
tried create environment -- same error

Aug 22 '16 16:08 teamschwarz

new output running docker 1.12 on a windows 10 box, using docker quickstart running as administrator:

$ ./adop compose gen-certs //root/.docker Generating client certificates for TLS-enabled Engine Loading 'screen' into random state - done Generating RSA private key, 4096 bit long modulus .......................................................................................................................................++ .............++ e is 65537 (0x10001) Loading 'screen' into random state - done Loading 'screen' into random state - done Signature ok subject=/CN=client Getting CA Private Key Could not read CA certificate "/c/Users/richard.a.schwarz/.docker/machine/certs/ca.pem": open /c/Users/richard.a.schwarz/.docker/machine/certs/ca.pem: The system cannot find the path specified.

Aug 23 '16 08:08 teamschwarz

Does "/c/Users/richard.a.schwarz/.docker/machine/certs/ca.pem" actually exist (both on the filesystem and from the perspective of the shell)?

What version of Docker Machine is in use as well please?

Aug 23 '16 10:08 nickdgriffin

docker 1.12. i worked with Anton on the phone, and the error message above was where were left off-temporarily stuck. i did check that the file was there, and had the correct contents. my docker terminal was launched as Admin. on a separate note: I accidentally closed the issue, so had to reopen it.

Aug 23 '16 15:08 teamschwarz

Docker 1.12 is the version of the Engine/Toolbox not Machine, but I'm going to install Docker Toolbox 1.12.1 on my Windows 10 PC at home and see if I can reproduce for a local machine (as it's not clear from the above what actual quickstart comand was run). I think this issue relates to Docker Machine though, as that's what creates the keys in question.

So, this is what I've done:

Installed Docker Toolbox 1.12.1 from fresh - my PC hasn't ever seen Docker before, so it's as clean an install as possible (immediately highlighting a difference in the fact that at least @teamschwarz upgraded from 1.11.x, and possibly the same for @anton-kasperovich too)
Pulled master for adop-docker-compose
Ran "./quickstart.sh -t local -u nick.griffin"
The machine came up fine and the output didn't state anything was wrong
The automatically triggered Create_Environment job executed fine
The pipeline completed successfully for all the automatic steps (I didn't bother with ProdA/B)

Versions:

Windows 10 Pro
Docker Toolbox 1.12.1
Docker version 1.12.1, build 23cf638
docker-machine.exe version 0.8.1, build 41b3b25
docker-compose version 1.8.0, build d988a55
GNU bash, version 4.3.46(2)-release (x86_64-pc-msys)

Output from docker info:

$ docker info
Containers: 24
 Running: 24
 Paused: 0
 Stopped: 0
Images: 21
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 467
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.17-boot2docker
Operating System: Boot2Docker 1.12.1 (TCL 7.2); HEAD : ef7d0b4 - Thu Aug 18 21:18:06 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 5.821 GiB
Name: adop
ID: YNPD:AZG4:NV5B:NWTM:YDLW:BIUK:AT3P:QKCD:37UW:ZIP4:FCFM:62GN
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 161
 Goroutines: 219
 System Time: 2016-08-24T20:50:08.377535378Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

Output from quickstart:

$ ./quickstart.sh -t local -u nick.griffin

      ###    ########   #######  ########
     ## ##   ##     ## ##     ## ##     ##
    ##   ##  ##     ## ##     ## ##     ##
   ##     ## ##     ## ##     ## ########
   ######### ##     ## ##     ## ##
   ##     ## ##     ## ##     ## ##
   ##     ## ########   #######  ##

Running pre-create checks...
Creating machine...
(adop) Copying C:\Users\Nick\.docker\machine\cache\boot2docker.iso to C:\Users\Nick\.docker\machine\machines\adop\boot2docker.iso...
(adop) Creating VirtualBox VM...
(adop) Creating SSH key...
(adop) Starting the VM...
(adop) Check network to re-create if needed...
(adop) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env adop

          ###    ########   #######  ########
         ## ##   ##     ## ##     ## ##     ##
        ##   ##  ##     ## ##     ## ##     ##
       ##     ## ##     ## ##     ## ########
       ######### ##     ## ##     ## ##
       ##     ## ##     ## ##     ## ##
       ##     ## ########   #######  ##

* Initialising ADOP
Sourcing provider-specific environment files...
Creating a new secrets file...
Your user name is nick.griffin
You have not provided a password. Generating random...
Your password is ***PASSWORD***
**Please make note of this as you will use this password to log into all the tools**
Generating random passwords for Jenkins, Gerrit and SQL...
Sourcing variables from platform.secrets.sh file...
* Setting up Docker Network
Created Docker network: local_network
* Pulling Docker Images
Sourcing provider-specific environment files...
Your secrets file already exists, moving on...
Sourcing variables from platform.secrets.sh file...
Pulling elasticsearch (elasticsearch:2.1.1)...
2.1.1: Pulling from library/elasticsearch
Digest: sha256:686b5409bb280b177476964754dee65a8d4ae76dcf9914690d001fd394e7ba42
Status: Image is up to date for elasticsearch:2.1.1
Pulling logstash (accenture/adop-logstash:0.1.0)...
0.1.0: Pulling from accenture/adop-logstash
Digest: sha256:215746631f1e88c68cfc124846d773081b03ef46e11234f027a6b5854165f52a
Status: Image is up to date for accenture/adop-logstash:0.1.0
Pulling kibana (kibana:4.3.1)...
4.3.1: Pulling from library/kibana
Digest: sha256:a46838c02553be07bdd7ef77eb14975e667e671aabe10f28d086b1c1ee3394f5
Status: Image is up to date for kibana:4.3.1

Pulling selenium-hub (selenium/hub:2.46.0)...
2.46.0: Pulling from selenium/hub
Digest: sha256:f58eae51c9c4038c0a92c1488c65581167c3c7b97430d39cf22d8210966956c3
Status: Image is up to date for selenium/hub:2.46.0
Pulling ldap-ltb (accenture/adop-ldap-ltb:0.1.0)...
0.1.0: Pulling from accenture/adop-ldap-ltb
Digest: sha256:82523c240fe1c276094f3cec284e8510e6c053266aabceb84a76cf03b6297e0f
Status: Image is up to date for accenture/adop-ldap-ltb:0.1.0
Pulling nexus (accenture/adop-nexus:0.1.0)...
0.1.0: Pulling from accenture/adop-nexus
Digest: sha256:19f57f44a9edb4383960888fbe8747ec864dd673c8044118d9f6a2efa9bf8ac1
Status: Image is up to date for accenture/adop-nexus:0.1.0
Pulling jenkins-slave (accenture/adop-jenkins-slave:0.1.4)...
0.1.4: Pulling from accenture/adop-jenkins-slave
Digest: sha256:5b6c0cbe299411a42e0eef0f303ef5585186e5411aa43761a69a07557ae51cc7
Status: Downloaded newer image for accenture/adop-jenkins-slave:0.1.4
Pulling gerrit-mysql (mysql:5.6.25)...
5.6.25: Pulling from library/mysql
Digest: sha256:47c62ff16fa535abe6c24844a0ce2b49b7d04e176422789fcf0d9fe566afbccb
Status: Downloaded newer image for mysql:5.6.25
Pulling ldap (accenture/adop-ldap:0.1.2)...
0.1.2: Pulling from accenture/adop-ldap
Digest: sha256:e132068ec2b6b7706f116cd9491656f0457366cb71a4cb10e956a022479ffd62
Status: Downloaded newer image for accenture/adop-ldap:0.1.2
Pulling gerrit (accenture/adop-gerrit:0.1.2)...
0.1.2: Pulling from accenture/adop-gerrit
Digest: sha256:32da9a444d98780238b6d95215b1430ee962593adbdd6508c12aa80f6e27c217
Status: Downloaded newer image for accenture/adop-gerrit:0.1.2
Pulling sonar-mysql (mysql:5.6.25)...
5.6.25: Pulling from library/mysql
Digest: sha256:47c62ff16fa535abe6c24844a0ce2b49b7d04e176422789fcf0d9fe566afbccb
Status: Image is up to date for mysql:5.6.25
Pulling sensu-api (accenture/adop-sensu:0.1.0)...
0.1.0: Pulling from accenture/adop-sensu
Digest: sha256:9ddb14de9b4f36b7b93a7580baa2bcfdb58b428193121d1b90c04473154f713f
Status: Downloaded newer image for accenture/adop-sensu:0.1.0
Pulling sonar (accenture/adop-sonar:0.2.0)...
0.2.0: Pulling from accenture/adop-sonar
Digest: sha256:1c4f3dbdfe665b6ac1b1f0248bf7732da9323b02789fdd9e3e6f1022eaf6bbd3
Status: Downloaded newer image for accenture/adop-sonar:0.2.0
Pulling sensu-client (accenture/adop-sensu:0.1.0)...
0.1.0: Pulling from accenture/adop-sensu
Digest: sha256:9ddb14de9b4f36b7b93a7580baa2bcfdb58b428193121d1b90c04473154f713f
Status: Image is up to date for accenture/adop-sensu:0.1.0
Pulling sensu-rabbitmq (rabbitmq:3.5.7-management)...
3.5.7-management: Pulling from library/rabbitmq
Digest: sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876
Status: Downloaded newer image for rabbitmq:3.5.7-management
Pulling selenium-node-chrome (selenium/node-chrome:2.46.0)...
2.46.0: Pulling from selenium/node-chrome
Digest: sha256:9ef27ace0fd081ae616a3d90fa8f6b937392f3ea58057d6f5fc011c27dcda5bb
Status: Downloaded newer image for selenium/node-chrome:2.46.0
Pulling proxy (accenture/adop-nginx:0.2.0)...
0.2.0: Pulling from accenture/adop-nginx
Digest: sha256:aadd6a0122a3930654653c7353e86dd7dc227b85a5a9440d5e37afb9d150a7e7
Status: Downloaded newer image for accenture/adop-nginx:0.2.0
Pulling sensu-redis (redis:3.0.7)...
3.0.7: Pulling from library/redis
Digest: sha256:a0a0d17b658bca52351da9a412429e28b3f35aca580ed298b4ddc96736add01c
Status: Downloaded newer image for redis:3.0.7
Pulling selenium-node-firefox (selenium/node-firefox:2.46.0)...
2.46.0: Pulling from selenium/node-firefox
Digest: sha256:f8229dd0cc0a87b7de8556653c636ac897acc4c0ef77d2c575ce4aff860e5fe7
Status: Downloaded newer image for selenium/node-firefox:2.46.0
Pulling sensu-uchiwa (sstarcher/uchiwa:0.15.0)...
0.15.0: Pulling from sstarcher/uchiwa
Digest: sha256:873ab60018947fdea70f5fd76f40c35a785ff635eb570ae22b9fa53fe4600111
Status: Downloaded newer image for sstarcher/uchiwa:0.15.0
Pulling jenkins (accenture/adop-jenkins:0.1.5)...
0.1.5: Pulling from accenture/adop-jenkins
Digest: sha256:dd5b5d32d1d1878adc2662b2b860307207a1d8e5695c71eada9ee618ea22e9b2
Status: Downloaded newer image for accenture/adop-jenkins:0.1.5
Pulling ldap-phpadmin (accenture/adop-ldap-phpadmin:0.1.0)...
0.1.0: Pulling from accenture/adop-ldap-phpadmin
Digest: sha256:dfa694f012b47875861c0098244cdbd5af751a6c3fee3e11431af144c9139b7d
Status: Downloaded newer image for accenture/adop-ldap-phpadmin:0.1.0
Pulling sensu-server (accenture/adop-sensu:0.1.0)...
0.1.0: Pulling from accenture/adop-sensu
Digest: sha256:9ddb14de9b4f36b7b93a7580baa2bcfdb58b428193121d1b90c04473154f713f
Status: Image is up to date for accenture/adop-sensu:0.1.0
* Bringing up ADOP...
Sourcing provider-specific environment files...
Your secrets file already exists, moving on...
Sourcing variables from platform.secrets.sh file...
Creating kibana
Creating elasticsearch
Creating logstash

Creating ldap-phpadmin
Creating nexus
Creating sensu-uchiwa
Creating selenium-hub
Creating sensu-api
Creating gerrit-mysql
Creating sonar
Creating sonar-mysql
Creating gerrit
Creating selenium-node-firefox
Creating ldap-ltb
Creating jenkins-slave
Creating ldap
Creating proxy
Creating sensu-redis
Creating sensu-server
Creating sensu-rabbitmq
Creating jenkins
Creating sensu-client
Creating selenium-node-chrome
* Waiting for the Platform to become available - this can take a few minutes
Jenkins was unavailable, so slept for: 30 secs
Jenkins was unavailable, so slept for: 30 secs
Jenkins was unavailable, so slept for: 30 secs
Jenkins was unavailable, so slept for: 30 secs
Jenkins was unavailable, so slept for: 30 secs
* Loading the Platform
Generating client certificates for TLS-enabled Engine
/c/Users/Nick/docker_certs/cert.pem was generated successfully...
/c/Users/Nick/docker_certs/ca.pem was generated successfully...
/c/Users/Nick/docker_certs/key.pem was generated successfully...
Uploading certificates to Jenkins Slave at: //root/.docker/
* Waiting for Nginx to become available
Nginx unavailable, sleeping for 5s
Nginx unavailable, sleeping for 5s
Nginx unavailable, sleeping for 5s
Nginx unavailable, sleeping for 5s

##########################################################

SUCCESS, your new ADOP instance is ready!

Run these commands in your shell:
  source ./conf/env.provider.sh
  source credentials.generate.sh
  source env.config.sh

You can check if any variables are missing with: ./adop compose config  | grep 'WARNING'

Navigate to http://192.168.99.100 in your browser to use your new DevOps Platform!
Login using the following credentials:
  Username: nick.griffin
  Password: ***PASSWORD***

After I tested the above, I also manually re-ran gen-certs as well:

$ ./adop compose gen-certs //root/.docker
Generating client certificates for TLS-enabled Engine
/c/Users/Nick/docker_certs/cert.pem was generated successfully...
/c/Users/Nick/docker_certs/ca.pem was generated successfully...
/c/Users/Nick/docker_certs/key.pem was generated successfully...
Uploading certificates to Jenkins Slave at: //root/.docker

So as far as I can see everything is fine, which makes me think either:

"something" was fixed between Docker Toolbox 1.12.0 and 1.12.1 (or rather, the versions of the other things it brings in) although there's nothing in the Machine release note to suggest they changed anything
or, it's specifically a problem when upgrading

Aug 24 '16 19:08 nickdgriffin

Hi, is this still an issue? If not, we could close it.

Oct 02 '19 15:10 deors

adop-cartridge-java adop-cartridge-java copied to clipboard

Create Environment fails using Docker Toolbox 1.12

new output running docker 1.12 on a windows 10 box, using docker quickstart running as administrator:

adop-cartridge-java
adop-cartridge-java copied to clipboard