dumbo
dumbo copied to clipboard
MultiMapper fails with single-parameter mappers
If I write a map function with the alternative low-level single-parameter interface, then give it to MultiMapper
:
import dumbo
from dumbo.lib import MultiMapper
from dumbo.decor import primary
@primary
def mapper(keysvals):
for key, val in keysvals:
yield key, val
if __name__ == "__main__":
multimapper = MultiMapper()
multimapper.add("primary", mapper)
dumbo.run(multimapper)
It tries to call it with two parameters:
...
File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 322, in run
File "build/bdist.linux-x86_64/egg/dumbo/util.py", line 49, in dumpcode
File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 314, in <genexpr>
File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 445, in mapfunc_iter
File "build/bdist.linux-x86_64/egg/dumbo/lib/__init__.py", line 180, in __call__
TypeError: mapper() takes exactly 1 argument (2 given)
It looks like MultiMapper
doesn't know about the alternative interface.
In general cases mapper callable object should support 2-parameters interface for ex.
def mapper(key, value):
yield key, value.split('\t')
Where value
is usually a single value or a line but not a generator.