Google Clould Builder fails app engine aetest

[roosevelt-junior]

Hi, I am migrating some unit tests that use aetest package to Google Cloud builder, but unfortunately the tests timeout:

=== RUN TestCreateUser
INFO 2018-09-08 02:53:50,929 devappserver2.py:281] Skipping SDK update check.
WARNING 2018-09-08 02:53:50,931 devappserver2.py:297] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
--- FAIL: TestCreateUser (15.00s)
main_test.go:25: timeout starting child process

main_test.go is:

func TestCreateUser(t *testing.T) {
	var w = httptest.NewRecorder()
	inst, err := aetest.NewInstance(&aetest.Options{StronglyConsistentDatastore: true})
	if err != nil {
		t.Fatal(err) // <----- LINE 25
	}
	defer inst.Close()

	req, err := inst.NewRequest("POST", `/api/profile`, nil)
	if err != nil {
		t.Fatal(err)
	}
	ctx := appengine.NewContext(req)
	req = req.WithContext(ctx)
	req.Header.Add("Authorization", user1)

	controllers.ProfileHandler(w, req)

	var users []*mdl.User
	q := ds.NewQuery("User").Filter("UserID =", user1)
	key, err := q.GetAll(ctx, &users)
	if err != nil {
		t.Errorf("User creation error, (want no error, got %v)", err)
	}
	if len(key) != 1 {
		t.Errorf("User creation error, (want: 1, got %v)", len(key))
	}
}

The Dockerfile is:

FROM mercari/appengine-go:latest

WORKDIR /app

ENV SRC_DIR=/go/src/github.com/auteurly/
# Add the source code:
ADD . $SRC_DIR
RUN cd $SRC_DIR; go get -v github.com/auteurly/...; go build -v github.com/auteurly/...; go test github.com/auteurly/app/app_engine -v

Note that if I deploy the image to GKE and ssh into the machine, I can successfully run the tests:

aakha@instance-1 ~ $ docker run gcr.io/auteurly-204321/backend-go-test:latest go test github.com/auteurly/app/app_engine -v
=== RUN   TestCreateUser
INFO     2018-09-08 03:20:42,033 devappserver2.py:281] Skipping SDK update check.
WARNING  2018-09-08 03:20:42,034 devappserver2.py:297] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING  2018-09-08 03:20:42,115 simple_search_stub.py:1196] Could not read search indexes from /tmp/appengine.testapp.root/search_indexes
INFO     2018-09-08 03:20:42,118 api_server.py:275] Starting API server at: http://localhost:35663
INFO     2018-09-08 03:20:42,122 dispatcher.py:270] Starting module "default" running at: http://localhost:34575
INFO     2018-09-08 03:20:42,123 admin_server.py:152] Starting admin server at: http://localhost:41539
--- PASS: TestCreateUser (5.36s)

[sbuss]

First guess is that cloudbuild is resource constrained so you need to increase the timeout. Try this:

inst, err := aetest.NewInstance(&aetest.Options{StronglyConsistentDatastore: true, StartupTimeout: 120 * time.Second})

[sbuss]

Edit: oh, we don't pass that timeout, and in fact don't even support it for standard. That 15s timeout is hardcoded...

Is this an App Engine Standard or Flex app? If it's Standard, there may not be anything we can do quickly.

[roosevelt-junior]

Thank you. Actually I have tried with higher timeouts which is explained by your reasoning above. This is App Engine standard.

I see that isntance_vm.go does override the StartupTimeout here: https://github.com/golang/appengine/blob/master/aetest/instance_vm.go#L38

Should we just switch to flex for testing?

[hovu96]

I am experiencing the same issue. Even using a bigger machine type doesn't solve the problem. Any idea how to troubleshoot this?

[roosevelt-junior]

I think the issue is that the machines's ports are not available for aetest to run the dev app server. I've had the same issue when running aetest cases locally while dev app server is running.

@sbuss can port collision be the root cause?

[broady]

Cloud builder gives you a dedicated VM that runs your build in a Docker container. The port should be available.

On Sun, Oct 14, 2018, 2:15 PM Ali Akhavan [email protected] wrote:

I think the issue is that the machines's ports are not available for aetest to run the dev app server. I've had the same issue when running aetest cases locally while dev app server is running.

@sbuss https://github.com/sbuss can aetest be run on a different port ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/golang/appengine/issues/159#issuecomment-429593028, or mute the thread https://github.com/notifications/unsubscribe-auth/AABhlrRYiq0Va7kCRWqf9D_sLJi_Vfqfks5ukqxggaJpZM4WiQ7G .

[johngb]

@nomadali I've had this problem for years when running a large number of tests with App Engine classic. The only solution that I've found is to rate limit the tests through a modified context. So when testing, my code is set up to call this NewContext() function instead of the normal one from the appengine.NewContext().

package myctx

import (
	"net/http"
	"time"

	"github.com/golang/protobuf/proto"

	"golang.org/x/net/context"
	"google.golang.org/appengine"
)

const (
	burstLimit = 100

	callsPerSecond = 750
)

var (
	throttledServices = map[string]bool{
		"datastore_v3": true,
	}

	throttle = make(chan time.Time, burstLimit)
)

func init() {
	ticker := time.NewTicker(time.Second / callsPerSecond)
	go func() {
		for t := range ticker.C {
			select {
			case throttle <- t:
			default:
			}
		}
	}()
}

func rateLimiter(ctx context.Context, service, method string,
	in, out proto.Message) error {
	if throttledServices[service] {
		<-throttle
	}
	return appengine.APICall(ctx, service, method, in, out)
}

func NewContext(req *http.Request) context.Context {
	ctx := appengine.NewContext(req)
	return appengine.WithAPICallFunc(ctx, rateLimiter)
}

Play with the burstLimit and callPerSecond values until it works. I've found that there is still some variability in the timeouts though, so if your throttling values are close to the limit for your tests and machine, it will sometimes work and sometimes fail. If that's the case just increase the throttling (lower burstLimit and callPerSecond values) and it should work.

[pbarjoueian]

@johngb Can you please share a working example of your solution? I mean an example which you are using this code. Thanks

[johngb]

@pbarjoueian To run any tests with the App Engine classic environment, one has to get a context first. Just use this NewContext() to get the context instead of appengine.NewContext()

If you want to know how to create local tests in general on App Engine Classic, the docs give a reasonable overview. In my case, I have a lot of code that is specific to my app in setting up the datastore and populating it, so I don't have anything clean to share.