Mapreduce tasks fail to start

[davidwtbuxton]

Hi,

I'm using the ReindexMapReduceTask, but it fails to start when using the local dev server, and when deployed to App Engine.

The first error is a TypeError exception:

Internal Server Error: /tasks/reindex/
Traceback (most recent call last):
  File "/Users/dave/myproject/sitepackages/prod/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/dave/myproject/sitepackages/prod/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/dave/myproject/bp/tasks/indexing.py", line 48, in reindex
    ReindexContentTask().start()
  File "/Users/dave/myproject/sitepackages/prod/djangae/contrib/mappers/pipes.py", line 162, in start
    pipe.with_params(target=self.target)
TypeError: get_deferred_target() takes no arguments (1 given)
ERROR    2017-02-17 18:29:39,936 base.py:284] Internal Server Error: /tasks/reindex/
Traceback (most recent call last):
  File "/Users/dave/myproject/sitepackages/prod/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/dave/myproject/sitepackages/prod/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/dave/myproject/bp/tasks/indexing.py", line 48, in reindex
    ReindexContentTask().start()
  File "/Users/dave/myproject/sitepackages/prod/djangae/contrib/mappers/pipes.py", line 162, in start
    pipe.with_params(target=self.target)
TypeError: get_deferred_target() takes no arguments (1 given)

And then after fixing that, it fails with a KeyError exception:

ERROR    2017-02-17 18:34:00,962 base.py:284] Internal Server Error: /tasks/reindex/
Traceback (most recent call last):
  File "/Users/myproject/sitepackages/prod/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/myproject/sitepackages/prod/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/myproject/myproject/tasks/indexing.py", line 48, in reindex
    ReindexContentTask().start()
  File "/Users/myproject/sitepackages/prod/djangae/contrib/mappers/pipes.py", line 163, in start
    pipe.start(base_path=PIPELINE_BASE_PATH, queue_name=queue_name)
  File "/Users/myproject/sitepackages/prod/mapreduce/pipeline_base.py", line 21, in start
    return pipeline.Pipeline.start(self, **kwargs)
  File "/Users/myproject/sitepackages/prod/pipeline/pipeline.py", line 673, in start
    self, idempotence_key, str(e)))
PipelineSetupError: Error starting djangae.contrib.mappers.pipes.DjangaeMapperPipeline(*('myproject.tasks.indexing.ReindexContentTask', 'djangae.contrib.mappers.thunks.thunk_map', 'djangae.contrib.mappers.readers.DjangoInputReader'), **{'params': {'_map': 'myproject.tasks.indexing.ReindexContentTask.map', 'args': (), 'db': 'default', 'kwargs': {}, 'model': 'myproject.Content', 'namespace': None}, 'shards': 1})#ea53b46c87f148f8bde2ee065eaa3182: KeyError('gLovneKdxm',)

I think the second error is because I used the wrong API to get the target. Should be using the module name, rather than the current version name.

[davidwtbuxton]

N.B. Using current head https://github.com/potatolondon/search/tree/78b63fda9756a598e9607aab9d61c084b108b75c

[davidwtbuxton]

Think the second error may be solved by this recent commit in the pipeline repo:

https://github.com/GoogleCloudPlatform/appengine-pipelines/commit/2f1930b0be21a40077676c6fc19a6b95801caa9e

Hasn't been released to PyPI yet, but you can install with pip like:

pip install --target libs https://github.com/GoogleCloudPlatform/appengine-pipelines/archive/master.zip#subdirectory=python/src

[davidwtbuxton]

For background see cl/126706214

Bloody Google open source.