Requests cancelled after 60 minutes

[paandahl]

trafficstars

Hello, I am using the jetty9-compat-image on Managed VMs, and have some issues with long-running requests. I am running an import, and after exactly 3.600 seconds it fails, and throws an error like this:

com.google.apphosting.api.ApiProxy$CancelledException: The API call search.IndexDocument() was explicitly cancelled.

The exact API call varies, depending on what is running at the time, but it's the same exception and error. My code spawns a background thread, and waits for it. It is the background thread that invokes the failing API call. My thread is spawned like this:

com.google.appengine.api.ThreadManager.currentRequestThreadFactory().newThread(myRunnable));

I can't find anything about a 60 minute deadline in any of the parts involved, am I missing something? From what I have read, there should be a 24 hour deadline for requests. Any idea what might be worng?

[paandahl]

I also tried to run the same piece of code without running it in a background thread, and encountered the same issue. After 3600s my request is cut off.

[lesv]

60 minutes is typically when your auth token expires, my guess you'll see something about that elsewhere in your logs. You need to refresh your token -- the client libraries should do that for you, but it sounds like you've found a case where they don't.

[lesv]

Taking a look at the JavaDocs it looks like you should call refreshToken() on your Credential.

[paandahl]

Hi, and thanks for the response. I have been browsing through the source code, but can't find any trace of Credential, or where to set / get it. As far as I understand, all the API calls go through com.google.apphosting.api.ApiProxy, but I can't find a way to set or refresh credentials on it.

[lesv]

@ludoch could you take a look at this.

I read through the source code and couldn't find anything that looked like a Credential, so it's probably on the "inside" portion of App Engine.

My guess, and I do not speak authoritatively in this case is that you are running up against a natural limit. Since you are using the Search api, the only work-a-round that I can suggest is that you break your process up into 58 minute blocks for now -- as that will create the correct credentials for you each time.

I expect something to be announced in two weeks that might help as well. Sorry I couldn't be of more help.

[ludoch]

Les, I do not see a limit set on the JVM process side. Please file an internal bug on the appserver bridge layer, I guess it is not Java specific.

On Sun, Mar 13, 2016 at 11:37 AM, Les Vogel [email protected] wrote:

@ludoch https://github.com/ludoch could you take a look at this.

I read through the source code and couldn't find anything that looked like a Credential, so it's probably on the "inside" portion of App Engine.

My guess, and I do not speak authoritatively in this case is that you are running up against a natural limit. Since you are using the Search api, the only work-a-round that I can suggest is that you break your process up into 58 minute blocks for now -- as that will create the correct credentials for you each time.

I expect something to be announced in two weeks that might help as well. Sorry I couldn't be of more help.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/200#issuecomment-196016764 .

[lesv]

Were you seeing an error 502?

[paandahl]

Yes, I probably should have mentioned that. The server returns a 502.

If the thread is not calling App Engine APIs it keeps running in the background though, but logging doesn't work as expected anymore (exceptions are being thrown from the log flusher).

[lesv]

Cool - it's a known issue.