Future intermittently missing on Heroku

[abulka]

When I deploy my Flask app to Heroku the Flask-Executor official example

@app.route('/start-task')
@app.route('/get-result')

which uses executor.submit_stored() to store the Future in the executor, the /get-result call behaves weirdly. It works fine on my development machine.

Sometimes executor.futures._state(job_id) returns None and sometimes it returns the correct state of the future e.g. RUNNING. Same long job run. It intermittently changes, as if sometimes the future is not found in the executor, and at other times it is found ok. On each call to the flask server, its a 50-50 toss up whether the future will be found or not.

Is there any way around this? some configuration perhaps? Perhaps I need to define a worker dyno? Perhaps I need to use RQ instead of Flask-executor? Does anyone understand what is going on? My flask app is deployed to Heroku using free dynos, using web gunicorn app:app in the Procfile.

[aaronyo]

Did you figure it out? If multiple python processes are running, then you likely have different Python processes servicing each request. Each Python process would have it's own instance of Flask-Executor. If it's literally a 50-50 tossup, suggests to me you have two python processes running.

[abulka]

I still can't figure out why each Python process would have it's own instance of Flask-Executor - I mean, all the other objects we create in our Python app are accessible ok - why is Flask-Executor particularly susceptible to this Heroku round robin process issue, I wonder.

Anyway, in my case, I removed Flask-Executor completely and went for a RQ and Redis solution. Runs on Heroku fine. Luckily the Heroku free plan allows one 'worker' process which picks off jobs from redis and runs them.

[dchevell]

I believe @aaronyo was right - and wouldn't be surprised if the "other objects" aren't actually sharing state between processes any more than this library. Solving via redis has nothing to do with Heroku and everything to do with using a shared memory store :)

Either way, sounds like you made the right choice for your needs here, I'll see about a doc update just to clarify that this doesn't do any inter-process data sharing.