scoop
scoop copied to clipboard
TypeError: cannot serialize 'greenlet.greenlet' object
For some reason, I'm getting an error because python is trying to serialize greenlets:
Traceback (most recent call last):
File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>
b.main()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 92, in main
self.run()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 290, in run
futures_startup()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
run_name="__main__"
File "/home/lochar/python/lib/python3.5/site-packages/scoop/futures.py", line 64, in _startup
result = _controller.switch(rootFuture, *args, **kargs)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_control.py", line 230, in runController
execQueue.sendResult(future)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_types.py", line 384, in sendResult
self.socket.sendResult(future)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_comm/scoopzmq.py", line 319, in sendResult
pickle.HIGHEST_PROTOCOL,
TypeError: cannot serialize 'greenlet.greenlet' object
I'm not using greenlets, so I'm not sure why I'm getting this. I'm really trying to get started with scoop—I've never used mapreduce nor done parallelized code before, so I might also be making an obvious error. Here is my code:
import os
import re
from scoop import futures as fut
def get_sideeffect_sentences(f):
sideeffect = re.compile(r"\bside\s+effects?\b")
tagsplit = re.compile(r"\s*<[hp]>\s*")
for ln in open(f):
for sent in tagsplit.split(ln):
if sideeffect.search(sent.lower()):
yield sent
def writeout(x, y):
print(y, file=outf)
if __name__ == "__main__":
scratch = os.getenv("SCRATCH")
outf = open(scratch + "/now/sideeffect_sents.txt", "w")
lsfiles = [ scratch + "/now/text/" + f for f in os.listdir(scratch + "/now/text") ]
fut.mapReduce(get_sideeffect_sentences, writeout, lsfiles)
outf.close()
The command is "python -m scoop -n 8". I noticed there are Québécois-sounding last names in the docs, so perhaps it might help to say I've been trying that on Briarée (Calcul Québec computer).
I am getting the same error