spinnaker-deploymentmanager
spinnaker-deploymentmanager copied to clipboard
GoogleCloudPlatform
Nothing appears on localhost:9000
IDK where else to report my issue with the GCP Cloud Launcher for Spinnaker. Hope people watching this project can point me in the right direction.
I used Google Cloud Launcher to make a new deployment of Spinnaker to evaluate. I used these values:
- Enabled the "Kubernetes" checkbox.
- Left alone the "Kubernetes Account Name" field (idk what value to use, and the help text doesn't answer that question).
- Set my GKE cluster's name in "Kubernetes Cluster Name" field - "cluster-mine-2".
- Set "Kubernetes Cluster Zone" to same zone as my fresh GKE 1.7 cluster.
- Left alone the "GCR Account Name" field.
When I SSH into that VM, my localhost:9000 doesn't respond.
$ gcloud compute ssh --project=$MY_PROJECT_ID --zone=$MY_ZONE spinnaker-1 -- -L9000:localhost:9000 -L8084:localhost:8084
Any ideas of what to change to make it work?
Hey @chexxor!
I'm not sure what could be going wrong but I do know that it takes about 5 minutes for Spinnaker to bootstrap fully when run from Cloud Launcher. Can you check the startups script log in /var/log/startupscript.log?
There's no /var/log/starupscript.log file.
chexxor@spinnaker-1:~$ ls /var/log
apache2 auth.log cloud-init.log dist-upgrade dmesg.0 kern.log ntpstats spinnaker udev upstart
apt boot.log cloud-init-output.log dmesg fsck landscape redis syslog unattended-upgrades
The cloud-init* logs don't have any errors. There is nothing in the gate and front50 logs, and I don't see any errors in the halyard logs.
Also, it's been up for 10 min, but localhost:9000 still doesn't respond. Which process handles requests on that port? I could check for it in top.
chexxor@spinnaker-1:~$ uptime
18:57:00 up 10 min, 1 user, load average: 0.00, 0.06, 0.07
Also, by "localhost:9000 doesn't response", I mean I get a page which says "This page isn’t working. localhost didn’t send any data.", and the HTTP request in the browser's network tab says it stalled out.
cc @lwander @GoogleCloudPlatform/spinnaker-google to see if they have any ideas on what could be wrong.
There should be an Apache service listening on 9000 afaik.
Looks like nothing is listening on port 9000, based on the commands I found from stackoverflow "what's listening on port?" search:
chexxor@spinnaker-1:~$ ss -lptn 'sport = :9000'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
chexxor@spinnaker-1:~$ sudo netstat -nlp | grep :9000
chexxor@spinnaker-1:~$ sudo lsof -n -i :9000 | grep LISTEN
Another clue - when I open the SSH tunnel to that VM using gcloud, I get this:
chexxor@spinnaker-1:~$ channel 5: open failed: connect failed: Connection refused
channel 6: open failed: connect failed: Connection refused
channel 5: open failed: connect failed: Connection refused
But I can still issue commands after those error messages.
I think I have a fix for this that will be public in ~10 minutes, I'll keep you posted.
Can you try again? If it still doesn't work you may not have an API enabled that new projects should by default. We'll figure out what that is & how to enable it if it doesn't work this time.
I spun up a new instance using the cloud launcher, same as I did before, but I get the same error. I think I made this GCP Project 8-10 months ago, if that helps you. Yeah, I was suspecting it would be an API enablement issue - Spinnaker is a complex system. When we figure that out, would be really nice for us to note those in the configuration wizard or "Get started with Spinnaker" section of the deployment page.
Jul 18 19:56:50 spinnaker-1 startup-script: INFO Starting startup scripts.
Jul 18 19:56:50 spinnaker-1 startup-script: INFO Found startup-script in metadata.
Jul 18 19:56:50 spinnaker-1 startup-script: INFO startup-script: Waiting for halyard to start running...
Jul 18 19:56:50 spinnaker-1 ec2:
Jul 18 19:56:50 spinnaker-1 ec2: #############################################################
Jul 18 19:56:50 spinnaker-1 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
...
Jul 18 19:56:51 spinnaker-1 ec2: -----END SSH HOST KEY FINGERPRINTS-----
Jul 18 19:56:51 spinnaker-1 ec2: #############################################################
Jul 18 19:56:59 spinnaker-1 startup-script: INFO startup-script: ERROR: (gcloud.service-management.enable) Request had insufficient authentication scopes.
Jul 18 19:56:59 spinnaker-1 startup-script: INFO startup-script: Return code 1.
Jul 18 19:56:59 spinnaker-1 startup-script: INFO Finished running startup scripts.
Looks like that's a great lead!
@skim1420 is this an expected IAM error? Seems perhaps that scopes are missing, while the API is already enabled.
I've just reproduced this problem. @chexxor can you please enable the service management API at https://console.cloud.google.com/apis/api/servicemanagement.googleapis.com/overview, and re-create a launcher instance?
I'll look into why this passed previously; perhaps legacy project issues.
The IAM issue appears to be resolved.
The new issue is one I feared would occur, because I didn't know how to choose an appropriate value for the GCR account and Kubernetes account.
Jul 18 21:32:51 spinnaker-1 startup-script: INFO startup-script: #015Problems in default.provider.dockerRegistry.my-gcr-account:
Jul 18 21:32:51 spinnaker-1 startup-script: INFO startup-script: ! ERROR Your docker registry has no repositories specified, and the registry's
Jul 18 21:32:51 spinnaker-1 startup-script: INFO startup-script: catalog is empty.
Jul 18 21:32:51 spinnaker-1 startup-script: INFO startup-script: ? Manually specify some repositories for this docker registry to index.
Jul 18 21:32:51 spinnaker-1 startup-script: INFO startup-script: - Failed to add account my-gcr-account for provider dockerRegistry.
Jul 18 21:32:52 spinnaker-1 startup-script: INFO startup-script: Return code 1.
I still can't access spinnaker on localhost:9000.
You will need to push at least one image to your docker registry
docker pull nginx
docker tag nginx:latest gcr.io/PROJECT_NAME/nginx
gcloud docker -- push gcr.io/PROJECT_NAME/nginx
will do the trick
Gotcha - I chose gcr.io as it was the default option and I expected the sub-host-less name to "just figure it out for me", so I didn't double-check that. My images are actually in us.gcr.io. :)
Next error:
ERROR Unable to communicate with your Kubernetes cluster: Operation: [list]
Unable to authenticate with your Kubernetes cluster
Full message from syslog:
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: - WARNING You have not specified a Kubernetes context in your halconfig,
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: Spinnaker will use "gke_$PROJECTID_$GOOGLEZONE_$GKECLUSTERNAME"
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: instead.
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: ? We recommend explicitly setting a context in your halconfig, to ensure changes
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: to your kubeconfig won't break your deployment.
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: ? Options include:
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: - gke_$PROJECTID_$GOOGLEZONE_$GKECLUSTERNAME
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: ! ERROR Unable to communicate with your Kubernetes cluster: Operation: [list]
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: for kind: [Namespace] with name: [null] in namespace: [null] failed..
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: ? Unable to authenticate with your Kubernetes cluster. Try using kubectl to
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: verify your credentials.
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: - I have just picked up a fault in the AE-35 unit.
Jul 18 23:55:56 spinnaker-2 startup-script: INFO startup-script: - Failed to add account my-kubernetes-account for provider kubernetes.
Jul 18 23:55:57 spinnaker-2 startup-script: INFO startup-script: Return code 1.
Oi vey - this is quite the chain of errors - what version is your k8s cluster on?
My GKE k8s cluster is latest - v1.7.0. I haven't been tracking the RBAC stuff in k8s - I wonder if that is breaking communication with it...
I haven't played with 1.7 at all, but I will be. I'll likely learn more about the state of its RBAC in the coming days. In the meantime, perhaps you've got some hypotheses.
Do you mean you haven't touched the RBAC settings yet? In that case I bet that the credentials you've downloaded work fine with kubectl, but fail to work with our client library since they are behind on auth support. I'm trying to replace our client with the more official kubernetes one: github.com/kubernetes-client/java. I'll let you know if I find a more immediate fix.
I haven't had any luck getting this to work via the web launcher https://console.cloud.google.com/launcher/details/click-to-deploy-images/spinnaker (it also doesn't seem to bring up a Jenkins instance as described in the docs)? I've tried a couple of times. I left Kubernetes and App Engine boxes default (unchecked).
I see something listening on port 8064 (looks like the Halyard API), but nothing on port 9000 (as an aside, there's an Apache instance listening on the default port that AFAICT isn't doing anything / forwarding to anything)?
I tried asking around on the Spinnaker Slack and didn't get any useful help, though it seems like other folks are seeing similar issues.
# netstat -anp | grep LISTEN
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1431/redis-server 1
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1820/sshd
tcp6 0 0 :::8064 :::* LISTEN 1415/java
tcp6 0 0 :::80 :::* LISTEN 2359/apache2
tcp6 0 0 :::22 :::* LISTEN 1820/sshd
Hey Will, sorry you're having such issues. Can you please verify that the Service Management API and Google Cloud Resource Manager API are enabled on your project?
Also, the Cloud Launcher Spinnaker doesn't include a Jenkins instance. You expecting a Jenkins instance leads me to believe maybe you're following along this codelab, which is not based on the Cloud Launcher?
Please let me know what you find, thanks.
@skim1420 I wasn't expecting one until I read the README in this very project (https://github.com/GoogleCloudPlatform/spinnaker-deploymentmanager), which I only found by searching for the filenames mentioned in the web interface.
I don't believe Service Management API is enabled, I guess I would have assumed that the launcher would warn me if something that was needed was not setup?
Ah, I see. Sorry about the confusion. The Cloud Launcher is actually something completely different than the DM template and setup in this current repository.
So you can either 1) follow the template here in this current repo, or 2) use the Cloud Launcher (which, again, has nothing to do with this current repo). For the latter, you can follow a tutorial, such as this one.
Depending on what you're trying to achieve/evaluate/learn, I can help point the way to get you started. Let me know.
@skim1420 thanks, and sorry also for the confusion. Tried Spinnaker slack per the launcher instructions, but didn't get much help there. I will try enabling the API, but re: #2, my understanding is that I shouldn't need to follow a tutorial to set things up if the launcher is working properly, no?
Correct, no need to follow a tutorial. I tend to point people there because there's unfortunately no real good place with the Launcher alone to ensure people enable the right APIs beforehand.
Re slack, sometimes the questions get swallowed if someone isn't immediately at hand. That's why we created the forum, where posts are longer-lived. Going forward, please try posting your questions there if you don't get immediate response on slack.
Thanks. FWIW, @skim1420 @viglesiasce, and sorry for the noise in a somewhat unrelated project (though since this is called deploy manager, as is the tool where I'm bringing up the test instance from, it seems like a reasonable mistake to make). Turning on service management API did resolve the issue; thanks for the help (this is not at all obvious to me, and doesn't seem to be mentioned in the page, so if you could point that out to the right folks, that might help avoid other people having the same problem).
Great to hear you're set for the time being. All of your feedback is reasonable, and actually our team (the Google Spinnaker team) owns the Cloud Launcher solution. There are limitations there we have to work with, but we'll keep at improving this as we can. Thanks.