dumbo icon indicating copy to clipboard operation
dumbo copied to clipboard

MultiMapper fails with single-parameter mappers

Open jkleint opened this issue 11 years ago • 1 comments

If I write a map function with the alternative low-level single-parameter interface, then give it to MultiMapper:

import dumbo
from dumbo.lib import MultiMapper
from dumbo.decor import primary

@primary
def mapper(keysvals):
    for key, val in keysvals:
        yield key, val

if __name__ == "__main__":
    multimapper = MultiMapper()
    multimapper.add("primary", mapper)
    dumbo.run(multimapper)

It tries to call it with two parameters:

...
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 322, in run
  File "build/bdist.linux-x86_64/egg/dumbo/util.py", line 49, in dumpcode
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 314, in <genexpr>
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 445, in mapfunc_iter
  File "build/bdist.linux-x86_64/egg/dumbo/lib/__init__.py", line 180, in __call__

TypeError: mapper() takes exactly 1 argument (2 given)

It looks like MultiMapper doesn't know about the alternative interface.

jkleint avatar Mar 18 '13 21:03 jkleint

In general cases mapper callable object should support 2-parameters interface for ex.

def mapper(key, value):
    yield key, value.split('\t')

Where value is usually a single value or a line but not a generator.

a4tunado avatar Apr 23 '13 21:04 a4tunado