workflow
workflow copied to clipboard
teamhephy
Deis app responds to health check but router returns 503 Service Temporarily Unavailable
From @olalonde on October 24, 2016 11:39
I just successfully deployed a new release with deis pull ... but when accessing the app from my browser, I am getting a 503 Service Temporarily Unavailable nginx error page.
The app logs show that it is responding to health check requests:
$ deis logs
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:19 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:29 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:39 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:49 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
$ deis config | grep PORT
PORT 5000
Some logs from deis router:
[2016-10-24T11:38:24+00:00] - myapp-ui - 172.20.0.83 - - - 503 - "GET / HTTP/1.1" - 792 - "http://myapp.com/" - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" - "myapp.com" - -- myapp.com - - - 0.000
[2016-10-24T11:38:29+00:00] - myapp-ui - 172.20.0.131 - - - 503 - "GET / HTTP/1.1" - 390 - "-" - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" - "myapp.com" - - - myapp.com - - - 0.000
[2016-10-24T11:38:52+00:00] - deis/deis-controller - 172.20.0.147 - - - 201 - "POST /v2/apps/myapp-ui/config/ HTTP/1.1" -2298 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 82.117 - 82.183
[2016-10-24T11:38:52+00:00] - deis/deis-controller - 172.20.0.131 - - - 200 - "GET /v2/apps/myapp-ui/config/ HTTP/1.1" - 1820 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 0.031 - 0.031
[2016-10-24T11:38:54+00:00] - deis/deis-controller - 10.244.0.1 - - - 200 - "GET /v2/apps/myapp-ui/config/ HTTP/1.1" - 1820 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 0.019 - 0.019
[2016-10-24T11:39:29+00:00] - myapp-ui - 172.20.0.147 - - - 503 - "GET / HTTP/1.1" - 390 - "-" - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" - "myapp.com" - - - myapp.com - - - 0.000
[2016-10-24T11:39:41+00:00] - myapp-ui - 172.20.0.83 - - - 503 - "GET /c/dJMPJb HTTP/1.1" - 390 - "-" - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - "myapp.com" - - - myapp.com - - - 0.000
But it doesn't show any of the request coming from my browser. So I'm guessing the deis router is having trouble with something. I tried to delete the deis-router pod but still same problem. Any idea?
Copied from original issue: deis/workflow#569
From @olalonde on October 24, 2016 11:55
Fixed by adding a Procfile with web process and deis ps:scale cmd=0 web=1
From @olalonde on October 24, 2016 20:8
@bacongobbler is this a new requirement? I thought it was fine to use cmd as the web process.
From @bacongobbler on October 24, 2016 20:12
Nope! It should be fine with cmd as the web process. I just closed it as I assumed it was an app issue with booting the process. Please re-open if you have more logs to share about how it failed to return a 200 OK with the cmd process
From @kmala on October 24, 2016 20:16
do you have a procfile with web type during the deis pull?
From @olalonde on October 24, 2016 20:37
I didn't have a Procfile when it failed. But it was the first deis pull for that app, previously I did git push deploys, maybe that's the reason? Maybe this should be tested in e2e tests (git push deploy followed by deis pull deploy)? I don't think it was a booting error because I could see the successful health check requests in my app logs. Anyways, I'm going to comment here if I have this issue again.
From @davidchua on April 8, 2017 8:5
I am having this issue as well.
I've tried with both having a Procfile with a cmd calling the same CMD in the dockerfile and also without a Procfile.
Application is running. Like @olalonde, previously I was git pushing from a buildpack before switching to a Dockerfile method.
From @jdotjdot on July 3, 2017 20:19
I'm also having this issue. Is there any resolution?
I was able to reproduce this after many hours. Please reopen for the record!
When a push with a Procfile/Heroku buildpack and then with Dockerfile happen on the same application, the service can reach a state 503 Service Unavailable. I had to recreate the app with a new name and then push there only with the relevant Dockerfile.
Seems like Procfile/Heroku Buildpack pushes can make the application have a dirty state.
The whole time I was trying to deploy for hours and thinking this was host-header problem with my application and the server hosting it inside the docker container. But it turns out the 503 Service Temporarily Unavailable is a problem with Deis.
From @jdotjdot on November 23, 2017 17:55
I would second reopening the issue. We literally have had to avoid using Deis because of it, and instead are using vanilla Kubernetes.
From @bacongobbler on November 23, 2017 18:2
This should've been fixed with https://github.com/deis/controller/pull/1201 and we closed this because it was a duplicate of #658. I'd take a look at that PR and see if there's some edge case missed there.
The code in question would be here: https://github.com/mboersma/controller/blob/master/rootfs/api/models/app.py#L478-L492
This is a nasty issue and we need to take a look at this one right away on the controller.
I was running into this. I had made some new services and forgotten to put EXPOSE in the dockerfile. I tried adding EXPOSE, which didn't help, I also tried renaming the process in the Procfile to "web", which also didn't work. I ended up just recreated the app in deis under a different name which fixed the issue.
@natethinks Thank you for reporting this. I will try to run some tests to see what happens when EXPOSE is not in the Dockerfile. Did you try setting the ENV variable PORT via the deis cli?
deis config:set PORT={your exposed docker port here}
This is might have helped as it would hopefully reset the PORT on the controller API to whatever port you specify there. The API defaults to exposing port 5000 so if you tried to expose any other port within your Dockerfile your app would have failed with 503 Service Temporarily Unavailable as the controller-api would not have been able to hit the healthcheck endpoint.