Issue with using recipy with python multiprocessing module

[ritviksahajpal]

Thanks for an awesome package! I wonder if recipy can work with python multiprocessing? The foll. example fails when recipy is imported, and works fine without recipy:

import recipy
import multiprocessing
import time

data = (
    ['a', '2'], ['b', '4'], ['c', '6'], ['d', '8'],
    ['e', '1'], ['f', '3'], ['g', '5'], ['h', '7']
)

def mp_worker((inputs, the_time)):
    print " Processs %s\tWaiting %s seconds" % (inputs, the_time)
    time.sleep(int(the_time))
    print " Process %s\tDONE" % inputs

def mp_handler():
    p = multiprocessing.Pool(2)
    p.map(mp_worker, data)

if __name__ == '__main__':
    mp_handler()

I get the foll. error:

Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Users\ritvik\Anaconda\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Users\ritvik\Anaconda\lib\site-packages\recipy\log.py", line 230, in log_exit
    db = open_or_create_db()
  File "C:\Users\ritvik\Anaconda\lib\site-packages\recipyCommon\utils.py", line 20, in open_or_create_db
    db = TinyDB(path, storage=serializer)
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\database.py", line 93, in __init__
    self._table = self.table(table)
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\database.py", line 110, in table
    table = self.table_class(StorageProxy(self._storage, name), **options)
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\database.py", line 202, in __init__
    data = self._read()
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\database.py", line 277, in _read
    return self._storage.read()
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb_serialization\__init__.py", line 139, in read
    data = self.storage.read()
  File "C:\Users\ritvik\Anaconda\lib\site-packages\tinydb\storages.py", line 101, in read
    return json.load(self._handle)
  File "C:\Users\ritvik\Anaconda\lib\json\__init__.py", line 291, in load
    **kw)
  File "C:\Users\ritvik\Anaconda\lib\json\__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "C:\Users\ritvik\Anaconda\lib\json\decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\ritvik\Anaconda\lib\json\decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 10072851 (char 10072850)

[robintw]

Thanks for reporting this.

It looks like it's an error within the tinydb module that we use for accessing our database. It is a very simple NoSQL database module, which is great for making recipy easy to install and get going with, but it means that it doesn't support various advanced features - and one of these is concurrent reading/writing from multiple processes or threads.

We have another open issue (#53) to deal with supporting multiple different types of database, including tinyDB and mongoDB - as this would also allow a number of other interesting usecases (eg. sharing recipy runs between users in a research group, for instance). I'm not sure exactly when we will get to this - it's not at the top of the priority list, but it's not too far down either (we're working on refactoring the CLI and sorting out automated testing first though).

[ritviksahajpal]

thanks @robintw !