bosh-init
bosh-init copied to clipboard
When using compiled releases, an incorrect stemcell sha leads to a panic
When updating stemcells for a bosh-init deployment, I often just update the stemcell version in the manifest and have bosh-init tell me what the sha1 should be, so I don't have to download the same file twice. This doesn't work when using compiled releases with bosh-init.
I use bosh-init version 0.0.81-775439c-2015-12-09T00:36:04Z
Steps to reproduce
- Use at least one compiled release in your bosh-init manifest, e.g.
releases:
- name: bosh
url: file://release-bosh-255.3-on-ubuntu-trusty-stemcell-3191.tgz
- Reference a stemcell with an incorrect sha1.
resource_pools:
- name: vms
network: private
stemcell:
url: https://bosh.io/d/stemcells/bosh-openstack-kvm-ubuntu-trusty-go_agent?v=3191
sha1: f3b0ed251700048c14c722cd995eb7d27e743fe9
Expected
An error message like this one
Downloading stemcell... Failed (00:01:17)
Failed validating (00:01:18)
Command 'deploy' failed:
Failed to download from 'https://bosh.io/d/stemcells/bosh-openstack-kvm-ubuntu-trusty-go_agent?v=3191':
SHA1 of downloaded file '3c520267dc7671d538d2e94ea9d382196ff1fca5' does not match expected SHA1 'f3b0ed251700048c14c722cd995eb7d27e743fe9'
Actual
bosh-init output is pretty silent
Started validating
Validating release 'bosh'... Finished (00:01:18)
Downloading release 'bosh-openstack-cpi'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
Validating cpi release... Finished (00:00:00)
Validating deployment manifest... Finished (00:00:00)
Downloading stemcell... Failed (00:00:39)
And the debug log shows a panic happening
File System] 2016/03/09 11:48:20 DEBUG - Checking if file exists /Users/cpi/.bosh_init/downloads/5bdaba97758737e8e606ff09beae1493fa150504-f3b0ed251700048c14c722cd995eb7d27e743fe9
[attemptRetryStrategy] 2016/03/09 11:48:20 DEBUG - Making attempt #0
[File System] 2016/03/09 11:48:20 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:32 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider056288482
[attemptRetryStrategy] 2016/03/09 11:48:32 DEBUG - Making attempt #1
[File System] 2016/03/09 11:48:32 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:45 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider556844249
[attemptRetryStrategy] 2016/03/09 11:48:45 DEBUG - Making attempt #2
[File System] 2016/03/09 11:48:45 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:59 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider683724900
[File System] 2016/03/09 11:48:59 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/bosh-init-release374149680
[File System] 2016/03/09 11:49:01 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/bosh-init-release575718863
[Main] 2016/03/09 11:49:01 ERROR - Panic: runtime error: invalid memory address or nil pointer dereference
********************
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/internal/github.com/cloudfoundry/bosh-utils/logger/logger.go:146 (0x9a57a)
/usr/local/go/src/runtime/asm_amd64.s:437 (0x5cd1e)
/usr/local/go/src/runtime/panic.go:423 (0x2dd09)
/usr/local/go/src/runtime/panic.go:42 (0x2c3c9)
/usr/local/go/src/runtime/sigpanic_unix.go:24 (0x42dc9)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deployment_preparer.go:169 (0x90786)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/ui/stage.go:65 (0xb955f)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deployment_preparer.go:182 (0x7a8f2)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deploy_cmd.go:74 (0x75ec0)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/runner.go:27 (0x8d60b)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/main.go:46 (0x284b)
/usr/local/go/src/runtime/proc.go:111 (0x301f0)
/usr/local/go/src/runtime/asm_amd64.s:1696 (0x5f041)
********************
@cppforlife any updates on this?
@voelzmo that's weird. we do that ourselves in our tests. can you show me stemcell.MF and release.MF?
Just to clarify, here is what I do:
- export a release for BOSH on some other director
- on another machine, write a deployment manifest, that
- uses the same Stemcell version referencing a URL on
bosh.io
- has an incorrect SHA1 in the manifest for the stemcell (because I have no way to obtain it on this machine)
- uses the same Stemcell version referencing a URL on
- then I do
bosh-init deploy
with that manifest
I would expect bosh-init
to tell me what the correct SHA1 should be to put in the manifest. That's what you do in your tests? Or do you use an already downloaded stemcell from the filesystem?
Anyways, here is my release.MF
---
compiled_packages:
- name: director
version: f40bfeb75a1aca2e4d0b89a2ed02d3f78bf2c19a
fingerprint: f40bfeb75a1aca2e4d0b89a2ed02d3f78bf2c19a
sha1: 70ddb007e4ee01692f54baf3ee51562421eb8e5c
stemcell: ubuntu-trusty/3215
dependencies:
- libpq
- mysql
- ruby
- s3cli
- name: genisoimage
version: 008d332ba1471bccf9d9aeb64c258fdd4bf76201
fingerprint: 008d332ba1471bccf9d9aeb64c258fdd4bf76201
sha1: 417b8dfcc0e6d8460f89263983f19617a8317a77
stemcell: ubuntu-trusty/3215
dependencies: []
- name: health_monitor
version: c305dc0bd463cc43b69f4365b786b2ba91173108
fingerprint: c305dc0bd463cc43b69f4365b786b2ba91173108
sha1: 8cf65bd65a777ac0e4f2c4b9ab65baaf262a6340
stemcell: ubuntu-trusty/3215
dependencies:
- ruby
- name: libpq
version: 09c8f60b87c9bd41b37b0f62159c9d77163f52b8
fingerprint: 09c8f60b87c9bd41b37b0f62159c9d77163f52b8
sha1: a5c433ff18e82840ca6c14fda116c71cded87d03
stemcell: ubuntu-trusty/3215
dependencies: []
- name: mysql
version: b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5
fingerprint: b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5
sha1: 23ad35455bed1a5afa8a431d903801fe60d22088
stemcell: ubuntu-trusty/3215
dependencies: []
- name: nats
version: 6a31c7bb0d5ffa2a9f43c7fd7193193438e20e92
fingerprint: 6a31c7bb0d5ffa2a9f43c7fd7193193438e20e92
sha1: fcfd7b3d5f9ec00d1bd1357bfaa541625fe707d5
stemcell: ubuntu-trusty/3215
dependencies:
- ruby
- name: nginx
version: 1d356bbd17ed8c349fd1053093222d78559687ec
fingerprint: 1d356bbd17ed8c349fd1053093222d78559687ec
sha1: 8104f109f912ae7adca1943f2a5f9efd8dce0ca3
stemcell: ubuntu-trusty/3215
dependencies: []
- name: postgres
version: 55dbf1828bbb8e5fcd7dc7550cfb73de89312458
fingerprint: 55dbf1828bbb8e5fcd7dc7550cfb73de89312458
sha1: bac0fe32bbda32523981f19d9d3040c208f974fa
stemcell: ubuntu-trusty/3215
dependencies: []
- name: postgres-9.4
version: 83e114e2fc524e9a7dfc27866f5d81e7c588642f
fingerprint: 83e114e2fc524e9a7dfc27866f5d81e7c588642f
sha1: 352a0e6fa4cc55926d54014c90592663e98a9cfe
stemcell: ubuntu-trusty/3215
dependencies: []
- name: powerdns
version: 256336d00b1689138490c385c03ad3a8f54b4a9e
fingerprint: 256336d00b1689138490c385c03ad3a8f54b4a9e
sha1: e7b9657fb422f516d61ebe6e934d4a0e11bc4f11
stemcell: ubuntu-trusty/3215
dependencies: []
- name: registry
version: e3a7ff24076d57208dc6ced1b64db057d6e36cc0
fingerprint: e3a7ff24076d57208dc6ced1b64db057d6e36cc0
sha1: c239e59109c29a9cf0b9c6eceb7029eefee64d90
stemcell: ubuntu-trusty/3215
dependencies:
- libpq
- mysql
- ruby
- name: ruby
version: 030e3d4eafd00b10bbfa879bcf44fba62139205e
fingerprint: 030e3d4eafd00b10bbfa879bcf44fba62139205e
sha1: dd57b4df07337c6bdf237f21f90b82cd82791ae6
stemcell: ubuntu-trusty/3215
dependencies: []
- name: s3cli
version: 939372842fa951cdc81440b62a8a325b561751f7
fingerprint: 939372842fa951cdc81440b62a8a325b561751f7
sha1: 3fa1c3bc5a7c616a6a2d992f87b3cf47b19ea508
stemcell: ubuntu-trusty/3215
dependencies: []
jobs:
- name: nats
version: 329a9df8c4b7d3655c57e59d8fe2a03d65843cac
fingerprint: 329a9df8c4b7d3655c57e59d8fe2a03d65843cac
sha1: 73de8659da15387295b879d8190c2d6d79037ac8
- name: blobstore
version: 076716f65051569ad5f3d45a80f233c5c8e5bcb0
fingerprint: 076716f65051569ad5f3d45a80f233c5c8e5bcb0
sha1: d436a508ceb731265db339b67bb1c4b95fe54791
- name: director
version: 108174c3d28e668f5150d1ee9ee0ccf38a674441
fingerprint: 108174c3d28e668f5150d1ee9ee0ccf38a674441
sha1: 0d01b1f187eb777f0170472e7312b743f1854254
- name: health_monitor
version: cfe20dfb6bd6daab58c55b38493c45120a557329
fingerprint: cfe20dfb6bd6daab58c55b38493c45120a557329
sha1: 7cd37a3fd5e6d80dabbc8a588c7a939845950b92
- name: postgres
version: 9ad09420dd12f830bfc54bb6891fca33e7ad4ff5
fingerprint: 9ad09420dd12f830bfc54bb6891fca33e7ad4ff5
sha1: 81ba89331da6c19e3847eb4baf4bdf2dd813799a
- name: postgres-9.4
version: 0c57c770101e1ee93da1ba2a32340fac1616e2cb
fingerprint: 0c57c770101e1ee93da1ba2a32340fac1616e2cb
sha1: 38d605b1be32860e8cceddfafd4a5bc356e93043
- name: powerdns
version: df20bcfaa20062ef2e0136f6b6a5dbdc62f7b6a8
fingerprint: df20bcfaa20062ef2e0136f6b6a5dbdc62f7b6a8
sha1: 6758d5bd60c6f21a852a42fac9ed0bd52915b748
- name: registry
version: cb87e2ee9c510212a3c28b106206f3b444980236
fingerprint: cb87e2ee9c510212a3c28b106206f3b444980236
sha1: 3401a86554db5e4c3ecc679ed7ef6b5171d30964
commit_hash: dff4913a
uncommitted_changes: true
name: bosh
version: 255.8+dev.1
The stemcell.MF
is probably coming directly from the archive on bosh.io, so I guess there shouldn't be a problem with this?