cloud sdk 123 causes PipelineSetupError in dev server

[snstanton]

I have a mapreduce pipeline that worked in SDK 122 that now fails in the dev server with the following error after updating to SDK 123:

Traceback (most recent call last): File "/vl/local/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch return method(_args, *_kwargs) File "/Users/stanton/src/vlshare/appserver/src/vl/base.py", line 645, in wrapper return handler(_args, *kwargs) File "/Users/stanton/src/vlshare/appserver/src/vlops/handlers.py", line 319, in any result = func(self.request) File "/Users/stanton/src/vlshare/appserver/src/vlops/admin/files.py", line 301, in getZone result = launchPipeline(GetZonePipeline(zone.key.urlsafe())) File "/Users/stanton/src/vlshare/appserver/src/vl/pipe_utils.py", line 62, in launchPipeline pipeline.start(base_path='/ops/pipeline') File "/Users/stanton/src/vlshare/appserver/src/lib/pipeline/pipeline.py", line 673, in start self, idempotence_key, str(e))) PipelineSetupError: Error starting vlops.admin.files.GetZonePipeline(('aghkZXZ-Tm9uZXIXCxIKWm9uZUVudGl0eRiAgICAgMCvCgyiAQZnbG9iYWw',), *{})#342c4ceab7084adfb389b35b45960fdd: As far as I can tell the pipeline is having problems submitting tasks to the task queue. Everything works as expected in the production environment, so this is specifically a problem with the dev server.

Any ideas? What additional logging would be helpful here?

[nbortolotti]

I believe I have a similar error:

ERROR 2016-11-15 18:25:09,247 webapp2.py:1552] Error starting views.MyPipeline((), *{})#6e0e7928be0e4ab9b11d93cddead70f2: Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in call rv = self.handle_exception(request, response, e) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in call rv = self.router.dispatch(request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher return route.handler_adapter(request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in call return handler.dispatch() File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch return self.handle_exception(e, self.app.debug) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch return method(_args, *_kwargs) File "/Users/nickbortolotti/CodeDevelopers/appengine/xworkflow/views.py", line 106, in post pipeline.start() File "/Users/nickbortolotti/CodeDevelopers/appengine/xworkflow/lib/pipeline/pipeline.py", line 673, in start self, idempotence_key, str(e)))

Ideas?

[przemekpastuszka]

I was able to fix that by modifying util.py:53 to: if pipeline._TEST_MODE or os.environ['SERVER_SOFTWARE'].startswith('Development'):

[blech]

@rtshadow could you please make that a pull request?

[cdman]

Hmm, might this be the same problem as #71? I think setting _TEST_MODE masks the issue and the proper fix would be to use the modules API as suggested in #71, rather than undocumented internal variables.