steady
steady copied to clipboard
eclipse/steady-rest-lib-utils crash with OutOfMemoryError
Hi, I deployed Steady on my computer with the docker images according to this guide [1]. Everything works fine within the first 30 mins to 1 hour. After that, when I check http://localhost:8033/haproxy?stats, I can see rest-lib-utils-nodes
is down.
however, the docker container is still running
When I check the log inside, I found it was crashed with OutOfMemoryError.
java.lang.OutOfMemoryError: Java heap space
at org.springframework.boot.loader.data.RandomAccessDataFile.read(RandomAccessDataFile.java:101) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parseEntries(CentralDirectoryParser.java:64) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:139) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:109) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:100) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.getRootJarFile(Handler.java:385) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.getRootJarFileFromUrl(Handler.java:373) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.openConnection(Handler.java:92) ~[rest-lib-utils.jar:3.2.0]
at java.base/java.net.URL.openConnection(Unknown Source) ~[na:na]
at javassist.JarClassPath.openClassfile(ClassPoolTail.java:163) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.ClassPoolTail.openClassfile(ClassPoolTail.java:340) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.ClassPool.openClassfile(ClassPool.java:621) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getClassFile3(CtClassType.java:217) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getClassFile2(CtClassType.java:178) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getModifiers(CtClassType.java:458) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.isInterface(CtClassType.java:443) ~[javassist-3.28.0-GA.jar!/:na]
at org.eclipse.steady.java.JarAnalyzer.getConstructIds(JarAnalyzer.java:501) ~[lang-java-3.2.0.jar!/:na]
at org.eclipse.steady.java.JarAnalyzer.getSharedConstructs(JarAnalyzer.java:702) ~[lang-java-3.2.0.jar!/:na]
at org.eclipse.steady.cia.rest.ArtifactController.intersect(ArtifactController.java:488) ~[classes!/:3.2.0]
at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
full log is here: steady-rest-lib-utils.log
Surprisingly, I can still process a scan even when steady-rest-lib-util
is down.
Then I tried to pull the newest steady git repo and up again. This time I did nothing but waiting. The steady-rest-lib-utils
crashed for the same reason within about 1.5 hours.
Could you please help me check why is this happening? Will this affect the functionality of the scan results(especially cve update and output)?
Regards.
[1] https://eclipse.github.io/steady/admin/tutorials/docker/#setup
Hello @zhaolida98
thank you for trying Steady and sorry for taking so long to come back to you.
The steady-rest-lib-util
service offers utility functionalities for libraries and it is not directly used during the scan of applications (for all goals but checkcode
). This is why you can still scan the application even when the steady-rest-lib-util
service is down or not working. That service is mainly used to establish whether libraries containing constructs modified to fix a vulnerability contain the vulnerable or fixed version of the code. Thus, when it's not working, you would get "orange hourglasses" in the results highlighting the facts that it is still unknown whether the dependencies contain the vulnerable or the fixed version of the code (See bottom of [2]).
To get more insights on what is going on, could you share the logs of steady-kb-importer
(that imports and updates the vulnerabilities) and steady-patch-lib-analyzer
(that invokes steady-rest-lib-util
for the task described above)?
Thank you!
[2] https://eclipse.github.io/steady/vuln_db/
OK I see, so when steady-rest-lib-util
is down, the scan will continue but the ability to detect vulnerability (lets just say CVEs ) are largely impaired, right? Anyway,
here is the log from `steady-patch-lib-analyzer`
2021-12-24 05:40:21,475 [pool-162-thread-4] [INFO ] tcheval.LibraryAnalyzerThread2 - ++++++++Thread 10 for library id [org.apache.hbase|hbase-shaded-client|3.0.0-alpha-1] finished+++++++++
2021-12-24 05:40:21,475 [pool-162-thread-4] [INFO ] tcheval.LibraryAnalyzerThread2 - Analysis of lib [[org.apache.hbase|hbase-shaded-client|2.4.7]] with tid [15]
2021-12-24 05:40:21,475 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/1.7.1?skipResponseBody=true&classifier=sources&packaging=jar]
2021-12-24 05:40:22,690 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.214 ms] (proxy=false)
2021-12-24 05:40:22,691 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/1.7.1/jar/constructIds/intersect?lang=JAVA, size=0.48 KB]
2021-12-24 05:41:08,212 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [200] in [45.521 ms] (proxy=false)
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - Qname [org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(InputStream,OutputStream)] not in sources of [[org.apache.hbase|hbase-shaded-client|1.7.1]]
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - Qname [org.apache.hadoop.hbase.ipc.SecureClient(Class,Configuration,SocketFactory)] not in sources of [[org.apache.hbase|hbase-shaded-client|1.7.1]]
2021-12-24 05:41:08,213 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7?skipResponseBody=true&packaging=jar]
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - cids contains [[null:JAVA|METH|org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(InputStream,OutputStream)]], change type [MOD]
2021-12-24 05:41:09,476 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.263 ms] (proxy=false)
2021-12-24 05:41:09,476 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7?skipResponseBody=true&classifier=sources&packaging=jar]
2021-12-24 05:41:10,598 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.121 ms] (proxy=false)
2021-12-24 05:41:10,599 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA, size=0.48 KB]
2021-12-24 06:11:10,208 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [500] in [000:29:59] (proxy=false)
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [X-Vulas-Version] = 3.2.0
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [X-Vulas-Component] = client
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [Content-Type] = application/json; charset=utf-8
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [Content-Language] = en-US
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Origin
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Access-Control-Request-Method
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Access-Control-Request-Headers
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Content-Type] = application/json
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Transfer-Encoding] = chunked
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Date] = Fri, 24 Dec 2021 06:11:10 GMT
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Connection] = close
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-body: [{"timestamp":1640326270207,"status":500,"error":"Internal Server Error","path":"/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect"}]
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Exception message: [Got error [500] when calling [POST] on [http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA]]
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - HTTP Request body: [[{"lang":"JAVA","type":"METH","qname":"org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(InputStream,OutputStream)","relates":null,"attributes":null},{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.ipc.SecureClient(Class,Configuration,SocketFactory)","relates":null,"attributes":null},{"lang":"JAVA","type":"METH","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(InputStream,OutputStream)","relates":null,"attributes":null}]]
2021-12-24 06:11:10,209 [pool-162-thread-4] [WARN ] tcheval.LibraryAnalyzerThread2 - The intersection returned null (thus something went wrong in cia); the Jar for library Id [[org.apache.hbase|hbase-shaded-client|2.4.7]] will not be included in the csv for MOD constructs
2021-12-24 06:11:10,209 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA, size=0.30 KB]
2021-12-24 06:13:40,560 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [500] in [000:02:30] (proxy=false)
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [X-Vulas-Version] = 3.2.0
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [X-Vulas-Component] = client
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [Content-Type] = application/json; charset=utf-8
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Request-header [Content-Language] = en-US
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Origin
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Access-Control-Request-Method
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Vary] = Access-Control-Request-Headers
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Content-Type] = application/json
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Transfer-Encoding] = chunked
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Date] = Fri, 24 Dec 2021 06:13:40 GMT
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-header [Connection] = close
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Response-body: [{"timestamp":1640326420559,"status":500,"error":"Internal Server Error","path":"/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect"}]
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - Exception message: [Got error [500] when calling [POST] on [http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA]]
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest - HTTP Request body: [[{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient(AuthMethod,Token,String)","relates":null,"attributes":null},{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient(AuthMethod,Token,String,boolean)","relates":null,"attributes":null}]]
2021-12-24 06:13:40,560 [pool-162-thread-4] [WARN ] tcheval.LibraryAnalyzerThread2 - The intersection returned null (thus something went wrong in cia); the artifact for library Id [[org.apache.hbase|hbase-shaded-client|2.4.7]] will not be included in the csv for ADD/DEL constructs
The real problem seems to locate in steady-rest-lib-util
. It shows an "out of memory" error. Before this error, there are plenty of First parameter in argument XXX does not match the to be skipped parameter [XX]
. Don't know what happened, I guess there is some memory leakage?
Here is the log from `steady-rest-lib-util`
``` 2021-12-24 05:57:31.396 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.collect.Tables$1()] 2021-12-24 05:57:34.550 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.io.Files$2()] 2021-12-24 05:57:34.559 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.io.Files$1()] 2021-12-24 05:57:50.204 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$3(CommonPattern)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.205 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$4(int)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.210 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$1(CharMatcher)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.211 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$2(String)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:58:31.427 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$4(long,int,OfLong,LongFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:31.428 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$5(long,int,OfDouble,DoubleFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.652 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$2(long,int,Iterator,FunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.652 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$3(long,int,OfInt,IntFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.653 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$1(long,int,Iterator,Iterator,BiFunction)] does not match the to be skipped parameter [Streams] 2021-12-24 06:00:06.078 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hadoop.hbase.shaded.org.apache.curator.framework.schema.SchemaSet$2(List,boolean)] does not match the to be skipped parameter [SchemaSet] 2021-12-24 06:02:14.705 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1()] 2021-12-24 06:02:58.680 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$4(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.681 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$3(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.682 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$2(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.682 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$1(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:04:58.204 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hadoop.hbase.shaded.org.apache.http.client.entity.GzipDecompressingEntity$1()] 2021-12-24 06:05:52.156 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.graph.Traverser$1(SuccessorsFunction,SuccessorsFunction)] does not match the to be skipped parameter [Traverser] 2021-12-24 06:05:52.157 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.graph.Traverser$2(SuccessorsFunction,SuccessorsFunction)] does not match the to be skipped parameter [Traverser] 2021-12-24 06:11:10.206 ERROR 8 --- [io-8092-exec-39] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root causejava.lang.OutOfMemoryError: Java heap space
at org.springframework.boot.loader.data.RandomAccessDataFile.read(RandomAccessDataFile.java:101) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parseEntries(CentralDirectoryParser.java:64) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.
2021-12-24 06:13:40.555 INFO 8 --- [nio-8092-exec-2] o.s.core.annotation.MergedAnnotation : Failed to introspect annotations on org.springframework.core.annotation.AnnotatedElementUtils$AnnotatedElementForAnnotations@290f8938: java.lang.OutOfMemoryError: Java heap space 2021-12-24 06:13:40.557 ERROR 8 --- [io-8092-exec-26] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8092-Acceptor" java.lang.OutOfMemoryError: Java heap space 2021-12-24 06:13:40.559 ERROR 8 --- [io-8092-exec-22] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Unknown Source) ~[na:na]
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) ~[na:na]
at java.base/java.lang.AbstractStringBuilder.append(Unknown Source) ~[na:na]
at java.base/java.lang.StringBuilder.append(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$LocaleDataStrategy.getCandidateLocales(Unknown Source) ~[na:na]
at java.base/sun.util.resources.Bundles.loadBundleOf(Unknown Source) ~[na:na]
at java.base/sun.util.resources.Bundles.of(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$1.run(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$1.run(Unknown Source) ~[na:na]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[na:na]
at java.base/sun.util.resources.LocaleData.getBundle(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData.getDateFormatData(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.initializeData(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.
2021-12-24 06:13:40.564 ERROR 8 --- [io-8092-exec-22] o.a.c.c.C.[Tomcat].[localhost] : Exception Processing ErrorPage[errorCode=0, location=/error]
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[tomcat-embed-core-9.0.46.jar!/:na] at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) ~[tomcat-embed-core-9.0.46.jar!/:na] at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.46.jar!/:na] at java.base/java.io.FilterOutputStream.flush(Unknown Source) ~[na:na] at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1193) ~[jackson-core-2.12.3.jar!/:2.12.3] at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1008) ~[jackson-databind-2.12.3.jar!/:2.12.3] ....
</details>
Correct, when it's down the ability to detect CVEs is impacted in terms of false positives (i.e., you may have CVEs reported with 'orange hourglasses' for dependencies already containing the fixed code). On top of it, it may also impact the import of CVEs: rest-lib-utils
is used during the import depending on the content of the statement.yaml
from https://github.com/SAP/project-kb. This is why I wanted to also have a look at the log from steady-kb-importer
to see whether the initial import worked. In fact, the first time you start the composition, the initial import of CVEs starts and that may imply an heavy usage of the steady-rest-lib-utils
service. Could you share the log of steady-kb-importer
?
I never experienced an "out of memory" on steady-rest-lib-util
running with the default memory options of Docker desktop on a machine with 16G, could you share the result in terms of memory (MEM USAGE / LIMIT) of docker stats
?
Sorry, I miss the log of steady-kb-importer
, it is surprisingly short here it is:
You have the latest version.
Fri Dec 24 05:07:59 UTC 2021 Kaybee Import already Running
no crontab for root
cron job created.
and my docker stats is:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
759f973115ad steady-patch-lib-analyzer 0.04% 421.4MiB / 15.66GiB 2.63% 22.6MB / 14.5MB 42.1MB / 20.5kB 27
ce25e92e0d59 steady-kb-importer 0.00% 16.12MiB / 15.66GiB 0.10% 238kB / 1.82kB 55.8MB / 12.3kB 2
3b54a11b6141 steady-haproxy 0.10% 12.42MiB / 15.66GiB 0.08% 14.4GB / 2.22GB 3.08MB / 0B 5
5186e310c9d1 steady-rest-backend 0.27% 1.515GiB / 15.66GiB 9.67% 25.1GB / 4.75GB 65.1MB / 0B 60
e89933a9fcaa steady-rest-lib-utils 0.04% 4.59GiB / 15.66GiB 29.30% 125MB / 9.56MB 198MB / 13.2MB 40
28a43af0ee0f steady-frontend-apps 0.10% 104.9MiB / 15.66GiB 0.65% 384MB / 5.57GB 4.7MB / 32.8kB 27
0b95cfddaadf steady-cache 0.00% 26.65MiB / 15.66GiB 0.17% 1.06GB / 1.21GB 5.61MB / 8.19kB 134
cbb1a24687b3 steady-frontend-bugs 0.10% 99.57MiB / 15.66GiB 0.62% 384MB / 4.84GB 8.04MB / 32.8kB 27
8f789168d712 steady-postgresql 0.00% 242MiB / 15.66GiB 1.51% 1.21GB / 23.8GB 7.8GB / 6.24GB 57
Just want to make sure, learned from previous issues[1], the kb-importer will automatically import the new CVEs every day, right?
[1]https://github.com/eclipse/steady/issues/500#issuecomment-916966158
@serenaponta Hi, did you figure out why is this problem coming? How to solve it?
Hi @zhaolida98 ,
your docker stats limits are in line with the ones of the containers we are running and, unfortunately, I was not able to reproduce the Out of Memory issue with rest-lib-utils. We are usually running it on Ubuntu 18.04 and I would recommend using Ubuntu - if possible and if it's not already the case.
From the logs of steady-kb-importer
I think the initial import of vulnerabilities from https://github.com/SAP/project-kb didn't complete successfully (even independently from rest-lib-utils). The initial import is time consuming and if it does not complete before the container is stopped, the import state remains "stuck" (there is a running
flag whose management needs to be improved).
I would suggest that you delete the folder /docker/kb-importer/data
and ensure that .env
contains KB_IMPORTER_SKIP_CLONE=True
. Upon the first startup, please plan at least 2h for the initial import with such flag (it takes ~2h30 while running "alone" on a machine with 8core and 16Gb of RAM). Once done the log of steady-kb-importer
looks as follows
Tue Jan 25 15:35:46 UTC 2022 Kaybee Import Done cron job created.
You can also check how many vulnerabilities are already imported during the import accessing the endpoint http://localhost:8033/backend/bugs
(500 vulnerabilities should be there once completed with KB_IMPORTER_SKIP_CLONE=True
).
Hi @serenaponta ,
Thanks for your reply. However, I checked http://localhost:8033/backend/bugs
, though the log is not expected, I can see 722 bugs there.
Actually, I am using Ubuntu 1804. In order to get rid of the outOfMemory
, now I decide to reinstall the steady using the stable version 3.2.2. I found three docker compose files docker-compose.build.yml
docker-compose-new.yml
and docker-compose.yml
, what is the difference between them? Which one should I use? I used to use docker-compose.yml
, is it correct?
Hi @zhaolida98,
from the number of vulnerabilities you have it looks like the flag KB_IMPORTER_SKIP_CLONE=True
in your .env
is not present, which is not a bad thing, it just means that the initial import takes far longer. The fact that the kb-importer state is now "stuck" can be seen also by the fact that it didn't import the newly published vulnerabilities as they are 727 as of today. In case you will reuse an existing data
folder in the new deployment, please remove the file docker/kb-importer/data/running
if present. (The new release will get rid of such problem.)
As for the docker compose files:
-
docker-compose.build.yml
is the one to use when building new docker images starting from the sources (e.g., if you want to build images for version 3.2.3-SNAPSHOT) as described at [1]. -
docker-compose.yml
is the one to start the composition using the docker images available in docker hub. -
docker-compose-new.yml
is a new compose file used by the scriptdocker/setup-steady.sh
that should make it easier to run steady and would allow to use different profiles to run only a subset of containers depending on the use case. As this is not yet documented, I would recommend you to continue usingdocker-compose.yml
.
[1] https://eclipse.github.io/steady/admin/tutorials/build/#building-docker-images-from-source
Closed due to lack of feedback from issue author