Deployment failing when using this plugin (container restarts 6 times)

[thepuzzlemaster]

I've got a node app which deploys fine to my staging server, but fails when deploying to my production server.

Staging is running dokku v0.22.9 Production is running dokku v0.25.1

I turned on tracemode and looked through both the staging logs and the production logs. I've found that the failure occurs here, where dokku sees 6 restarts in my docker container.

Waiting for 10 seconds ...
remote: + is_container_status ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311 Running
remote: + declare 'desc=return 0 if given docker container id is in given state'
remote: + local CID=ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + local 'TEMPLATE={{.State.Running}}'
remote: ++ docker container inspect --format '{{.State.Running}}' ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + local CONTAINER_STATUS=true
remote: + [[ true == \t\r\u\e ]]
remote: + return 0
remote: ++ docker container inspect --format '{{ .RestartCount }}' ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + local container_restarts=6
remote: + [[ 6 -ne 0 ]]
remote: + docker container update --restart=no ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + docker container stop ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + dokku_log_fail 'App container failed to start!!'
remote: + declare 'desc=log fail formatter'
remote: + echo ' !     App container failed to start!!'
remote:  !     App container failed to start!!
remote: + exit 1
remote: + checks_check_deploy_cleanup ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + declare 'desc=cleans up CHECK_DEPLOY_TMP_WORK_DIR and print container output'
remote: + local id=ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311
remote: + rm -rf /tmp/dokku-29385-trigger-scheduler-docker-local-check-deploy.17BXNo
remote: + [[ -n ef9328b1d5336a5ebd84eb38413aabe7f235557d8cb1c21af385f0071195a311 ]]
remote: + dokku_log_info2_quiet 'myApp web container output:'
remote: + declare 'desc=log info2 formatter (with quiet option)'
remote: + [[ -z '' ]]
remote: + echo '=====> myApp web container output:'

I looked through my changes to see what I might have added to cause the failure and was able to narrow it down to the inclusion of this plugin.

When I rename my apt-packages file, the deployment works fine to my production server.

So I suppose I mostly just have 2 questions. 1: Any bells ringing around why the exact same code would push fine to my staging server, but fail on my production server? The differing dokku versions is the most obvious difference I can think of. Could it be a memory issue on the production server? 2: Where in the huge debug output could I look for maybe a bit more info on why my container is restarting 6 times? I tried searching for exit status 1, but I'm not seeing anything stand out when I do that.

I'm going to try upgrading my staging server to the same version of dokku running on production (I didn't realize these were running different versions) and see if that makes a difference.

If i can't get anywhere, I could sanitize my log output of sensitive info and post that, but I thought I see if anyone had any thoughts as to what might be going on before doing that.

Thanks.

[thepuzzlemaster]

Alright, I upgraded my staging server to the latest version of dokku, and with no other changes, it then failed as well. So it works fine using dokku v0.22.9 Fails using dokku v0.25.1 and v0.26.6

My apt-packages file looks like:

libgtk2.0-0
libgtk-3-0
libgbm-dev
libnotify-dev
libgconf-2-4
libnss3
libxss1
libasound2
libxtst6
xauth
xvfb

I suppose I'll start removing packages from the file until it stops failing.

[thepuzzlemaster]

So it would appear the problem is with libnotify-dev. When I remove that package, the build succeeds fine.

[josegonzalez]

Are both the staging and prod server using the same version of herokuish? Seems like maybe thats the issue.

[thepuzzlemaster]

Neither server recognizes herokuish version when run from the command line.

They both report: herokuish: command not found

Is there a different way to check this?

[josegonzalez]

You can get the version via dokku report.

[thepuzzlemaster]

Gotcha. Here's the current state of things:

Staging

• Dokku: 0.26.6
• Herokuish: 0.5.33

Production

• Dokku: 0.25.1
• Herokuish: 0.5.30

However after having removed the libnotify-dev entry from my apt-packages file, it succeeds on both servers.

Previously the libnotify-dev package installed fine and didn't cause any build issues with dokku version 0.22.9. But I upgraded that server and can't check what version of herokuish was running on it.

I could probably try to put together a minimal repro if that'd be helpful. I'm not sure if it's the libnotify-dev package by itself, or if it has something to do with my specific build.

[josegonzalez]

Apologies for the late reply.

TBH this sounds like the version of libnotify being used isn't the one you should be using for this app. We normally have container logs for the failing container after the message myApp web container output:, so that would be a good starting ground.

The only other thing I can think of is the OS being used in herokuish is different (your staging was newer) and you were using herokuish-18 vs herokuish-20 in one or the other. Without a sample app, I can't really debug further, but if you can provide one, I'm happy to investigate.

Closing as there isn't much to do here.