deploy on Heroku

[lagleki]

I wish there was a fast method of deploying Kassy on Heroku.

[drkno]

Potentially something that could be contributed? Currently there is docker which is a starting point. It would be good to support, but isn't something we can currently prioritise.

[matthewhartstonge]

For this to work, there doesn't seem to be much change needed albeit - assuming it just works on heroku. It looks like package.json needs to be updated with an engine, and a web script added. So you should end up with something like the following, notice the change in engines and scripts.

{
  "name": "kassy",
  "version": "3.5.1",
  "description": "Extensible general purpose chat bot.",
  "main": "main.js",
  "dependencies": {
    "babel-plugin-transform-strict-mode": "^6.3.13",
    "babel-polyfill": "^6.3.14",
    "babel-preset-es2015": "^6.3.13",
    "babel-register": "^6.3.13",
    "babylon": "^6.8.0",
    "chalk": "^1.1.3",
    "cline": "^0.8.2",
    "colors": "^1.1.2",
    "deasync": "^0.1.4",
    "facebook-chat-api": "1.1.0",
    "figlet": "^1.1.1",
    "fs-extra": "^0.26.3",
    "monkey-patches-node-telegram-bot-api": "0.0.1",
    "node-telegram-bot-api": "^0.21.1",
    "npm": "^3.7.5",
    "redwrap": "*",
    "request": "^2.72.0",
    "rimraf": "^2.4.4",
    "sandbox": "^0.8.6",
    "sanitize-filename": "^1.5.3",
    "skyweb": "0.0.2",
    "stripcolorcodes": "^0.1.0",
    "tmp": "^0.0.28",
    "ws": "^0.8.0"
  },
  "devDependencies": {
    "chai": "^3.5.0",
    "grunt": "latest",
    "grunt-contrib-watch": "^1.0.0",
    "grunt-mocha-cli": "^2.1.0",
    "grunt-run": "^0.6.0",
    "mocha": "^2.4.5"
  },
  "engines": {
    "node": "4.4.5"
  },
  "scripts": {
    "test": "grunt test",
    "start": "node main.js",
    "web": "node main.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/mrkno/Kassy.git"
  },
  "author": "Matthew Knox",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/mrkno/Kassy/issues"
  },
  "homepage": "https://github.com/mrkno/Kassy"
}

[lagleki]

Can anyone confirm that Telegram gatewaying is working?

[drkno]

@lagleki you should ask @phillipadsmith

[phillipadsmith]

@lagleki Yes, it is working as far as I can tell, i.e., the bot I'm running is up on Telegram (more so that Skype!)

[drkno]

Skype is one of those pet problems.... Should have a SkypeBot integration soon to fix the problem permanently.

[phillipadsmith]

SkypeBot

Can you say more about this? Very curious...

[drkno]

Basically a WIP of https://developer.microsoft.com/en-us/skype/bots

[phillipadsmith]

Oh, those Skype bots! Sweet. :-)

[lagleki]

@phillipadsmith Skype integration doesn't work on Heroku? But why?

[drkno]

@lagleki its more a matter of Skype in general being broken everywhere thanks to #133 and other upstream regressions. Hopefully when the SkypeBots branch has landed we can forget about this issue.

[phillipadsmith]

@lagleki Yep, what @mrkno said. :-)

[parasdsingh]

@MatthewHartstonge I tried deploying to heroku using the package.json edit. Unfortunately it didn't work. So I proceeded on with the recommended Procfile.

web: node main.js facebook

PS: Logging into Facebook is pain. They keep blocking Heroku servers for security purposes

The application runs but with a few issues.

• During startup, after modules are loaded I see the following error

2016-10-08T17:18:13.340358+00:00 app[web.1]: Loading module 'update'...     [DONE]
2016-10-08T17:18:13.347076+00:00 app[web.1]: fatal: Not a git repository (or any parent up to mount point /app)
2016-10-08T17:18:13.347078+00:00 app[web.1]: Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2016-10-08T17:18:14.143608+00:00 app[web.1]: Starting integrations...

• 60 seconds after startup, I see the following error which crashes it.

2016-10-08T17:19:04.025109+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch

I realize this one might be fixed by changing the Procfile to

worker: node main.js facebook

but I'm unsure since Concierge need to listen to HTTP traffic via the integrations. I've seen a patch for this by adding a stub express.js GET listener.

Any suggestions?

[drkno]

@ParasDPain @MatthewHartstonge even if you do get it to launch on Heroku, the filesystem will be readonly (which means no persistent configuration, no module installation/uninstallation). Potentially you could partially get around the problem by committing a config and the modules (Hubots approach) or using the CLI options to point the module directory to /tmp which is r/w but it still isn't ideal.

[parasdsingh]

@mrkno Thanks for the heads-up! I might look into alternative solutions

[drkno]

@ParasDPain if you come up with a good solution to the problem, do tell. It would be nice to be able to deploy to Heroku while still having the benifits of Concierge.

[matthewhartstonge]

Hey team, I've been away for the past couple of weeks hence the late reply. Personally, I haven't played with Heroku myself, just had a quick check of their docs. A worker dyno seems to be the one you want, as otherwise it will complain a port isn't open. Reading into it seems to claim there isn't much difference between the two types though - as you've noticed the web type monitors and checks that the app is serving requests. Have you had any further luck?

[lagleki]

https://github.com/finnp/gitter-irc-bot - Gitter-IRC bridge works in Heroku. But yeah, we need another open port for Telegram.

2016-10-11 2:23 GMT+03:00 Matt Hartstonge [email protected]:

Hey team, I've been away for the past couple of weeks hence the late reply. Personally, I haven't played with Heroku myself, just had a quick check of their docs. A worker dyno seems to be the one you want, as otherwise it will complain a port isn't open. Reading into it seems to claim there isn't much difference between the two types though - as you've noticed the web type monitors and checks that the app is serving requests. Have you had any further luck?

On Mon, 10 Oct 2016 at 14:19 Matthew Knox [email protected] wrote:

@ParasDPain https://github.com/ParasDPain if you come up with a good solution to the problem, do tell. It would be nice to be able to deploy to Heroku while still having the benifits of Concierge.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/concierge/Concierge/issues/168# issuecomment-252525933>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ AF1gfVmibSbT9p9kYvnXUmAFtgPxAACKks5qyZKNgaJpZM4IxKUR>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/concierge/Concierge/issues/168#issuecomment-252771988, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTREX3wqlWsLLCDd_r4XE7d_l9nxHvHks5qysjZgaJpZM4IxKUR .

[parasdsingh]

Currently, the Procfile is not being detected by Heroku and I don't know why and I've managed to utilize package.json to modify how Concierge is ran (If no Procfile is detected, heroku defaults to calling npm start) I modified the start script in package.json to run with my facebook integration.

It's still running as a web task. Will update as I fix things

[parasdsingh]

I've pinpointed the issue about why Procfile wasn't detected. ~~The node-buildpack that comes with heroku has web as it's primary configuration~~ I think Heroku sets web as primary configuration for all apps and if the Procfile is missing a web declaration, it is ignored altogether (details here).

This can be resolved by manually scaling web down to 0 with

heroku scale web=0 worker=1 --app APP_NAME

As far as I know this has to be done manually after creating the heroku app and can't be done inside the Procfile which sucks. At the end of the day, scaling down is still an ugly patch because the Heroku router still tries connecting to the app and pollutes the log file with this

2016-10-11T08:32:10.656724+00:00 app[worker.1]: info Got answer in 1079 2016-10-11T08:32:16.486398+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=<redacted>.herokuapp.com request_id=35782ab8-efa1-4aa4-9046-177adcc96e7c fwd="54.161.76.89" dyno= connect= service= status=503 bytes= 2016-10-11T08:32:21.039076+00:00 app[worker.1]: info Got answer in 10382 2016-10-11T08:32:21.208052+00:00 app[worker.1]: info Got answer in 169

A potential solution might be providing Concierge's console output optionally out on a webpage as well i.e. Concierge runs a GET listener on a port, though I don't see any applications of this beside making heroku happy. On a sidenote, it would be nice to be able to run Concierge in the browser alone with something like Browserify but I suspect it'll require extensive refactoring of code and structure

I've resolved the port issue; persistence and GIT_DISCOVERY_ACROSS_FILESYSTEM issues remain.

[parasdsingh]

@mrkno I just noticed all modules are installed in /tmp directory by default for r/w access and I didn't had to specify it manually. So it does partially persist right outta the box.

[drkno]

@ParasDPain it clones them there before moving them to modules. If it is working from there out of the box then its magic and don't question it (but I approve :)).

Are modules that require npm to install something working?

[parasdsingh]

@mrkno You are right. They are indeed being moved to /app/modules after cloning into /tmp.

And no, modules fail to install if they require some module from npm :(

2016-10-13T02:22:19.904340+00:00 app[worker.1]: Cloning into '/tmp/tmp-3jWqBTga0dkY3'... 2016-10-13T02:22:20.019267+00:00 app[worker.1]: Loading module 'evil'... Installing "/app/modules/kpm_evil/evil.js" from npm. 2016-10-13T02:22:22.245811+00:00 app[worker.1]: [FAIL]

[parasdsingh]

I'm wondering if using the dyno's config vars for Concierge's persistence will be a good idea. The variables remain persistent across restarts and can be accessed via process.env. A simple wrapper to set/get them should do the job

Though there is a size limit of 32kb for the vars per dyno which certainly puts a limit on how many modules can be used at once.

Alternatives:

• Committing persistent data (modules/services) to heroku's fork of the source code immediately after any change has been made to the files
• Using a third party storage service like S3 and update it before restarts (might have to make this periodic as I'm not sure how much control we get over the app while heroku is restarting it)

[parasdsingh]

@mrkno I've decided to write a service that makes a git commit to heroku's local git repo every time a file is changed. Still WIP here.

A problem now is installing and configuring a remote integration for testing and just normal use case. AFAIK heroku does not provides access to the apps CLI. I'm considering writing an auto-deployment service that runs user-defined commands on the terminal but am having trouble accessing Concierge's stdout programmatically. It's needed to gain access to kpm commands, so I could call something like /kpm install facebook programmatically.

Any suggestions?

[drkno]

Passing messages to platform.onMessage(api, event) will in turn pass them to kpm down the line. The event object can be generated using shim (accessible anywhere. You could also use this for api, but I would suggest just reusing an existing api object if you can). I had considered deploying Concierge using tmux which would allow a detached console to run in the background which could then be attached to using ssh.

Note that even if Heroku now has a r/w filesystem, any writes are temporary and will be deleted whenever Heroku moves your deploy to a different server (happens frequently). Additionally, your application may be restarted, unloading things like configuration which may be subsequently lost.... (the problems go on, if only they provided us with either persistent storage or a way to mount an in memory FS).

[parasdsingh]

Yikes! Looks like the mountain of problems in heroku are far too many. I might stop pursuing a persistent build if the current plan doesn't works out.

Thanks for the tips! I'd prefer using the Concierge's internal API as well 👍

[drkno]

Basically, Heroku is designed for static apps, or ones that depend purely on a database and/or S3 for storage. Our use case is "do anything you want" which only partially fits within their vision. I don't know if Heroku + Docker does the same as normal AWS (custom container with everything we want) but it may be one workaround for this problem. There was some work done with beta.0 to make it so that configuration can be saved to a DB getting around that persistence problem, but we may end-up having to build an in-memory filesystem on-top of a database if you want the full features of KPM :(

If you are willing to take the hubot route (full AOT config and setup, no KPM like features) then Concierge already works out of the box with Heroku with no problems.

[parasdsingh]

I feel it'd be much easier and painless to instead use AWS or DigitalOcean 🤔

Building an in-memory FS is only advantageous for restricted PaaS like heroku. Would it be useful anywhere else? I'm sure there are better features we can work on for the next version.

UPDATE: Giving up on persisting using a git-commit service (for the time being)

[Awarua-]

Progress for Heroku with the PostgreSQL DB module https://github.com/concierge/postgresql