Asynchronous Task Queues in Python

Several implementations of asynchronous task queues in Python using the multiprocessing library and Redis.

Blog post: Developing an Asynchronous Task Queue in Python

Setup

1. Fork/Clone

2. Create and activate a virtual environment

3. Install the dependencies

4. Enter the Python shell and download the NLTK stopwords corpus:

   >> import nltk
   >> nltk.download('stopwords')
   
     [nltk_data] Downloading package stopwords to
     [nltk_data]     /Users/michael.herman/nltk_data...
     [nltk_data]   Unzipping corpora/stopwords.zip.
     True
   

Examples

Multiprocessing Pool:

$ python simple_pool.py

Multiprocessing Queue:

$ python simple_queue.py
$ python simple_task_queue.py

Logging to a single file:

$ python simple_task_queue_logging.py

Logging to separate files:

$ python simple_task_queue_logging_separate_files.py

Redis:

$ python redis_queue.py