Using Win_worker with Flask

[kc1]

I develop locally on win10, which is a problem for the usage of the RQ task queue, which only works on linux systems because it requires the ability to fork processes. I'm trying to extend the flask-base project https://github.com/hack4impact/flask-base/tree/master/app which can use RQ. I came across https://github.com/michaelbrooks/rq-win . I love the idea of this repo (If I can get it working it will really simplify my life, since I develop on win 10 -64):

After installing this library

I can queue a job in my views by running something like:

     @login_required
     @main.route('/selected')
     def selected():
     messages = 'abcde'
      j = get_queue().enqueue(render_png, messages, result_ttl=5000)
      return j.get_id()

This returns a job_code correctly.

I changed the code in manage.py to:

from rq_win import WindowsWorker

@manager.command
def run_worker():
    """Initializes a slim rq task queue."""
    listen = ['default']


    REDIS_URL = 'redis://localhost:6379'
    conn = Redis.from_url(REDIS_URL)

    with Connection(conn):
        # worker = Worker(map(Queue, listen))
        worker = WindowsWorker(map(Queue, listen))
        worker.work()

When I try to run it with:

$ python -u manage.py run_worker 


09:40:44
09:40:44 *** Listening on ?[32mdefault?[39;49;00m...
09:40:58 ?[32mdefault?[39;49;00m: ?[34mapp.main.views.render_png('{"abcde"}')?[39;49;00m (8c1b6186-39a5-4daf-9c45-f60e4241cd1f)
...\lib\site-packages\rq\job.py:161: DeprecationWarning: job.status is deprecated. Use job.set_status() instead
  DeprecationWarning
09:40:58 ?[31mValueError: Unknown type <class 'redis.client.StrictPipeline'>?[39;49;00m
Traceback (most recent call last):
  File "...\lib\site-packages\rq_win\worker.py", line 87, in perform_job
    queue.enqueue_dependents(job, pipeline=pipeline)
  File "...\lib\site-packages\rq\queue.py", line 322, in enqueue_dependents
    for job_id in pipe.smembers(dependents_key)]
  File "...\lib\site-packages\rq\queue.py", line 322, in <listcomp>
    for job_id in pipe.smembers(dependents_key)]
  File "...\lib\site-packages\rq\compat\__init__.py", line 62, in as_text
    raise ValueError('Unknown type %r' % type(v))
ValueError: Unknown type <class 'redis.client.StrictPipeline'>

So in summary, I think the jobs are being queued correctly within redis. However when the worker process tries to grab a job off of the queue to process, This error occurs. How can I fix this?

[michaelbrooks]

Looking at the source code for the worker class built in to rq (https://github.com/rq/rq/blob/master/rq/worker.py), it seems the interface for workers has changed substantially. I last tested this with rq v0.3, and it is now on 0.10. I suggest you try downgrading your rq library to see if that helps. Alternatively, it may be possible to upgrade this extension to support the new library.

[kc1]

Michael, I'm not an expert python programmer, but if you have any insights on how to approach this I'd be willing to play with it . Meanwhile I will try to downgrade to rq 0.3

[kc1]

Downgraded to rq 0.3.13:

$ rqworker -w rq_win.WindowsWorker
15:33:17 Using rq_win.WindowsWorker (experimental)
15:33:17 RQ worker started, version 0.3.13
15:33:17
15:33:17 *** Listening on ?[32m?[39;49;00m...
15:33:51 ?[32mdefault?[39;49;00m: ?[34mapp.main.views.render_png('{MY ARGUMENTS}', result_ttl=5000)?[39;49;00m (21dc12d8-0c96-459c-9025-ee3ff74e0429)
Traceback (most recent call last):
  File "...\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "...\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "...\Scripts\rqworker.exe\__main__.py", line 9, in <module>
  File "...\lib\site-packages\rq\scripts\rqworker.py", line 90, in main
    w.work(burst=args.burst)
  File "...\lib\site-packages\rq_win\worker.py", line 42, in work
    return super(WindowsWorker, self).work(burst)
  File "...\lib\site-packages\rq\worker.py", line 326, in work
    self.fork_and_perform_job(job)
  File "...\lib\site-packages\rq\worker.py", line 373, in fork_and_perform_job
    child_pid = os.fork()
AttributeError: module 'os' has no attribute 'fork'

[michaelbrooks]

Hmm I'm not sure why downgrading didn't work. os.fork is only available on unix systems, not on Windows. The point of the rq-win extension was to avoid having to call os.fork.

[kc1]

Hi Stephen,

Thanks for looking at this. Attached is a stripped down version of what I have so far.

You can login to the flask-app with

[email protected] 12345678

I'm running the worker through the manage.py file using git bash

$ python -u manage.py run_worker

I appreciate you taking a quick look if you have time.

https://drive.google.com/file/d/1pGq_BhYmTi2z4UYvhDrfF6nL-cb7g3Mj/view

[aytey]

@kc1 I just opened-up PR5, which hopefully might resolve this for you (at least, I had very similar issues as you were facing, and things now work for me)!