webarchive-indexing
webarchive-indexing copied to clipboard
Building a local cluster fails with ValueError
python build_local_zipnum.py zipnum ~/Downloads/*.cdx.gz
Traceback (most recent call last):
File "build_local_zipnum.py", line 102, in <module>
main()
File "build_local_zipnum.py", line 98, in main
run_job(r.inputs, r.output, r.shards, r.parallel, r.numlines)
File "build_local_zipnum.py", line 38, in run_job
job = ZipNumClusterJob(args)
File "/home/anand/.virtualenvs/py27/local/lib/python2.7/site-packages/mrjob/job.py", line 78, in __init__
super(MRJob, self).__init__(self.mr_job_script(), args)
File "/home/anand/.virtualenvs/py27/local/lib/python2.7/site-packages/mrjob/launch.py", line 109, in __init__
self.configure_options()
File "/home/anand/projects/sandbox/webarchive-indexing/zipnumclusterjob.py", line 42, in configure_options
help='Number of lines per gzipped block')
File "/home/anand/.virtualenvs/py27/local/lib/python2.7/site-packages/mrjob/launch.py", line 431, in add_passthrough_option
self.add_passthru_arg(*args, **_optparse_kwargs_to_argparse(**kwargs))
File "/home/anand/.virtualenvs/py27/local/lib/python2.7/site-packages/mrjob/options.py", line 1596, in _optparse_kwargs_to_argparse
raise ValueError('invalid option type: %r' % kwargs['type'])
ValueError: invalid option type: <type 'int'>