appengine

appengine copied to clipboard

I've noticed that my task workers never run in tests, and there's an almost complete de-coupling of enqueue/dequeue logic in testing that prevents me from even verifying that the task was properly enqueued.

Consider the following minimal example:

app.yaml:

runtime: go
api_version: go1

handlers:
 - url: /worker/.*
   script: _go_app
   login: admin
 - url: /.*
   script: _go_app

package worker

func SettleWorker(w http.ResponseWriter, r *http.Request) {
	ctx := appengine.NewContext(r)
	log.Infof(ctx, "Worker was successfully invoked")
}

package service

func TestHandler(w http.ResponseWriter, r *http.Request) {
	ctx := appengine.NewContext(r)
	t := taskqueue.NewPOSTTask("/worker/wrongpath", map[string][]string{"name": {"BLAH"}})
	task, err := taskqueue.Add(ctx, t, "")
	if err != nil {
		http.Error(w, "Error", http.StatusBadRequest)
		return
	}
	log.Infof(ctx, "Successfully posted task to worker: %+v", task)
}

func init() {
	http.HandleFunc("/worker/settle", worker.SettleWorker)
	http.HandleFunc("/test", TestHandler)
}

package service_test

func TestTestHandler(t *testing.T) {
	inst, err := aetest.NewInstance(nil)
	if err != nil {
		t.Fatal(err.Error())
	}
	req, err := inst.NewRequest("GET", "/", nil)
	if err != nil {
		t.Fatal(err.Error())
	}
	resp := httptest.NewRecorder()
	http.HandlerFunc(service.TestHandler).ServeHTTP(resp, req)
}

Note that the path being passed to taskqueue.Add isn't a valid path. This causes no errors when I test this using goapp test bitwyrm/service -v - I see the happy logging statement and a status code of 200 is returned from TestHandler. The logging statement from SettleWorker does not appear. The same thing happens if the path is correct.

In both cases, there's logging output that suggests that the taskqueue stub is failing to actually find the worker endpoint:

INFO     2018-05-03 09:51:11,794 module.py:846] default: "POST /worker/settle HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:11,794 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.100 seconds
INFO     2018-05-03 09:51:11,897 module.py:846] default: "POST /worker/settle HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:11,897 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.200 seconds
INFO     2018-05-03 09:51:12,101 module.py:846] default: "POST /worker/settle HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:12,101 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.400 seconds

When I run this outside of test using dev_appserver.py, the correct path does trigger the worker, and the incorrect path triggers 403 responses with a series of retries (both behaviors you'd expect).

Why is this behavior happening? As far as I know this is a behavior that is unique to Go - are other App Engine Standard runtimes are able to find the relevant task queue workers when testing?

In any event, this behavior is frustrating - ideally we'd be able to run tests that actually trigger task workers in order to test a given workflow end-to-end.

I've pushed up a small demo repository that you can use to replicate this error locally: https://github.com/venantius/demo