spring-cloud-dataflow
spring-cloud-dataflow copied to clipboard
spring-cloud
spring-cloud-dataflow rest client api 504 gateway timed out
I have 2 applications deployed in ECS ( openshift ). One is spring cloud dataflow server second is spring cloud dataflow rest client.
I was able to create app registry , create task and launch task via rest client api application.But after I upgraded to latest dataflow rest client 2.9.2 or 2.9.4 , it is giving me timed out 504 exception while creating task via rest api client methods.
details are below, while creating task , it is calling below rest url with size=2000. This url will get timedout with 2000 even in postman also. But I give size=20 in postman, it is working fine. You can see size=2000 was defined in restclient jar as below.
http://dataflow-server/tasks/definitions?size=2000
This is exact line to create task by passing task_name and register app_name.
Task.builder(dfo).name(task_Name).definition(app_Name).description(taskDesc).build();
below rest client jar class
TaskTemplate.java
@Override
public TaskDefinitionResource.Page list() {
String uriTemplate = definitionsLink.getHref();
uriTemplate = uriTemplate + "?size=2000";
return restTemplate.getForObject(uriTemplate, TaskDefinitionResource.Page.class);
}
What version of the data flow server are you using? Is there any log/stack trace on the server that you can share? The 504 sounds like an erroneous http code, something else is going on there.
Hi markpollack, Thanks for replying this thread first.
Both data flow server and data flow rest client is 2.9.4. Even I have tried with latest version 2.9.5 version but no luck.
Please find the below log stack.
2022-08-24 15:42:48.120 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : HTTP POST http://wr3-scdf-server.net/apps/task/wr3-processing-app-232-v4/1.0.RCA.232
2022-08-24 15:42:48.121 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2022-08-24 15:42:48.121 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : Writing [{uri=[docker://docker.artifactrepository/wr3-processing-smc:1.0.RCA.232], metadata-uri=[], force=[true]}] with org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter
2022-08-24 15:42:48.139 DEBUG 18588 --- [nio-8080-exec-5] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader@690c9ab67 pairs: {POST /apps/task/wr3-processing-app-232-v4/1.0.RCA.232 HTTP/1.1: null}{Accept: application/json, application/*+json}{Content-Type: application/x-www-form-urlencoded;charset=UTF-8}{User-Agent: Java/11.0.14}{Host: wr3-scdf-server.net}{Connection: keep-alive}{Content-Length: 153}
2022-08-24 15:42:48.570 DEBUG 18588 --- [nio-8080-exec-5] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader@344f018e4 pairs: {null: HTTP/1.1 201}{Content-Length: 0}{Date: Wed, 24 Aug 2022 10:12:48 GMT}{Set-Cookie: 670f4b959a4aae9002433ce2627bd02d=a41437ce05aef255d4b583bbfe21ad3b; path=/; HttpOnly}
2022-08-24 15:42:48.570 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : Response 201 CREATED
2022-08-24 15:42:48.571 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : HTTP GET http://wr3-scdf-server.net/tasks/definitions?size=2000
2022-08-24 15:42:48.573 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2022-08-24 15:42:48.574 DEBUG 18588 --- [nio-8080-exec-5] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader@435d4d8c5 pairs: {GET /tasks/definitions?size=2000 HTTP/1.1: null}{Accept: application/json, application/*+json}{User-Agent: Java/11.0.14}{Host: wr3-scdf-server.net}{Connection: keep-alive}
2022-08-24 15:43:18.600 DEBUG 18588 --- [nio-8080-exec-5] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader@76575a2a5 pairs: {null: HTTP/1.1 504 Gateway Time-out}{Content-length: 92}{Cache-Control: no-cache}{Connection: close}{Content-Type: text/html}
2022-08-24 15:43:18.601 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.client.RestTemplate : Response 504 GATEWAY_TIMEOUT
2022-08-24 15:43:18.611 ERROR 18588 --- [nio-8080-exec-5] c.c.r.c.ScdfOrchestrationController : 504 Gateway Time-out: [no body]
org.springframework.web.client.HttpServerErrorException$GatewayTimeout: 504 Gateway Time-out: [no body]
at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:116) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.cloud.dataflow.rest.client.VndErrorResponseErrorHandler.handleError(VndErrorResponseErrorHandler.java:62) ~[spring-cloud-dataflow-rest-client-2.9.5.jar:2.9.5]
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:819) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:777) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:334) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.cloud.dataflow.rest.client.TaskTemplate.list(TaskTemplate.java:148) ~[spring-cloud-dataflow-rest-client-2.9.5.jar:2.9.5]
at org.springframework.cloud.dataflow.rest.client.TaskTemplate.list(TaskTemplate.java:53) ~[spring-cloud-dataflow-rest-client-2.9.5.jar:2.9.5]
at com.ScdfOrchestrationController.createTask(ScdfOrchestrationController.java:95) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.19.jar:5.3.19]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.19.jar:5.3.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) ~[tomcat-embed-core-9.0.62.jar:[4.0.FR](http://4.0.fr/)]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.19.jar:5.3.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.62.jar:[4.0.FR](http://4.0.fr/)]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.19.jar:5.3.19]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2022-08-24 15:43:18.620 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Req: null CReq: null RP: null Stage: - Thread: http-nio-8080-exec-5 State: N/A Method: Constructor URI: N/A
2022-08-24 15:43:18.620 DEBUG 18588 --- [nio-8080-exec-5] org.apache.coyote.AsyncStateMachine : Changing async state from [DISPATCHED] to [STARTING]
2022-08-24 15:43:18.621 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Firing onStartAsync() event for any AsyncListeners
2022-08-24 15:43:18.622 DEBUG 18588 --- [nio-8080-exec-5] o.s.w.c.request.async.WebAsyncManager : Started async request
2022-08-24 15:43:18.622 DEBUG 18588 --- [nio-8080-exec-5] o.s.w.c.request.async.WebAsyncManager : Async result set, dispatch to /scdf/createTask
2022-08-24 15:43:18.623 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Req: 36755095 CReq: e27097c RP: 441767a5 Stage: 3 Thread: http-nio-8080-exec-5 State: N/A Method: dispatch URI: /scdf/createTask
2022-08-24 15:43:18.624 DEBUG 18588 --- [nio-8080-exec-5] org.apache.coyote.AsyncStateMachine : Changing async state from [STARTING] to [MUST_DISPATCH]
2022-08-24 15:43:18.625 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.servlet.DispatcherServlet : Exiting but response remains open for further handling
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@65e9bb55:org.apache.tomcat.util.net.NioChannel@21711a15:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57857]], Status in: [OPEN_READ], State out: [LONG]
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] org.apache.coyote.AsyncStateMachine : Changing async state from [MUST_DISPATCH] to [DISPATCHING]
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@65e9bb55:org.apache.tomcat.util.net.NioChannel@21711a15:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57857]], State after async post processing: [ASYNC_END]
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] o.a.c.authenticator.AuthenticatorBase : Security checking request POST /scdf/createTask
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] org.apache.catalina.realm.RealmBase : No applicable constraints defined
2022-08-24 15:43:18.627 DEBUG 18588 --- [nio-8080-exec-5] o.a.c.authenticator.AuthenticatorBase : Not subject to any constraint
2022-08-24 15:43:18.628 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Req: 36755095 CReq: e27097c RP: 441767a5 Stage: 3 Thread: http-nio-8080-exec-5 State: N/A Method: intDispatch URI: /scdf/createTask
2022-08-24 15:43:18.628 DEBUG 18588 --- [nio-8080-exec-5] org.apache.coyote.AsyncStateMachine : Changing async state from [DISPATCHING] to [DISPATCHED]
2022-08-24 15:43:18.634 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.servlet.DispatcherServlet : "ASYNC" dispatch for POST "/scdf/createTask", parameters={}
2022-08-24 15:43:18.635 DEBUG 18588 --- [nio-8080-exec-5] s.w.s.m.m.a.RequestMappingHandlerAdapter : Resume with async result [<400 BAD_REQUEST Bad Request,504 Gateway Time-out: [no body],[]>]
2022-08-24 15:43:18.648 DEBUG 18588 --- [nio-8080-exec-5] o.s.w.s.m.m.a.HttpEntityMethodProcessor : Using 'application/json', given [application/json] and supported [text/plain, /, text/plain, /, application/json, application/+json, application/json, application/+json]
2022-08-24 15:43:18.649 DEBUG 18588 --- [nio-8080-exec-5] o.s.w.s.m.m.a.HttpEntityMethodProcessor : Writing ["504 Gateway Time-out: [no body]"]
2022-08-24 15:43:18.660 DEBUG 18588 --- [nio-8080-exec-5] o.s.web.servlet.DispatcherServlet : Exiting from "ASYNC" dispatch, status 400
2022-08-24 15:43:18.661 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Firing onComplete() event for any AsyncListeners
2022-08-24 15:43:18.663 DEBUG 18588 --- [nio-8080-exec-5] o.apache.catalina.core.AsyncContextImpl : Req: 36755095 CReq: e27097c RP: 441767a5 Stage: 3 Thread: http-nio-8080-exec-5 State: N/A Method: recycle URI: /scdf/createTask
2022-08-24 15:43:18.664 DEBUG 18588 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@65e9bb55:org.apache.tomcat.util.net.NioChannel@21711a15:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57857]], Status in: [OPEN_READ], State out: [CLOSED]
2022-08-24 15:43:18.664 DEBUG 18588 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@65e9bb55:org.apache.tomcat.util.net.NioChannel@21711a15:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57857]], Status in: [OPEN_READ], State out: [CLOSED]
2022-08-24 15:43:18.664 DEBUG 18588 --- [nio-8080-exec-5] o.apache.tomcat.util.threads.LimitLatch : Counting down[http-nio-8080-exec-5] latch=1
2022-08-24 15:43:18.664 DEBUG 18588 --- [nio-8080-exec-5] org.apache.tomcat.util.net.NioEndpoint : Calling [org.apache.tomcat.util.net.NioEndpoint@3bae6249].closeSocket([org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@65e9bb55:org.apache.tomcat.util.net.NioChannel@21711a15:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57857]])
while task creation , this below url was called internally from data flow rest client jar.
http://wr3-scdf-server.net/tasks/definitions?size=2000
and it got timed out with size of 2000.
The same url got timed out in post man with same error. Exactly after 30 seconds it is getting timed out.
But if i give size=20 in postman then it is working.
In this case though, it isn't that the response is slow, it returns an error so I think it is something different. We need to reproduce.
The 504 from the gateway implies the gateway timeout is lower than 180s which is the default in Tomcat. Looks like they may be using Spring Cloud Gateway with 30s timeout.
It may be time to fix paging on anything that may return more than 100 items. We should consider setting limits on page sizes where appropriate.
Yes, this does seem sort of related to #5055. Classifying this as a high priority bug to investigate.
We have seen this before regarding under provisioned database machines. Likely your container for the database is under resourced. We will do a sanity check on our end wrt to the query.
Any info you have on the infrastructure supporting the server and DB would be useful, e.g. selected size.