preparing siard for browse - error not displayed

[Laurira]

I loaded a database successfully and clicked "Browse". Then it started preparations for browsing and it keeps doing that about 24 hours. No error is displayed to the user. If I check logs then I see 2021-08-10 11:48:32,318 [http-nio-8080-exec-7] ERROR c.d.c.s.controller.SIARDController - Conversion to SIARD failed for database 2f23f8c9-4230-4c56-b3f9-2cdf692bc2e4 org.roda.core.data.exceptions.GenericException: Could not convert the database. at com.databasepreservation.common.server.controller.SIARDController.convertSIARDtoSolr(SIARDController.java:713) at com.databasepreservation.common.server.controller.SIARDController.loadFromLocal(SIARDController.java:669) at com.databasepreservation.common.api.v1.CollectionResource.createCollection(CollectionResource.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392) at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:385) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:560) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:501) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:438) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: Invalid format: "\u0000007-09-27" at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) at org.joda.time.DateTime.parse(DateTime.java:160) at com.databasepreservation.utils.JodaUtils.xsDateParse(JodaUtils.java:98) at com.databasepreservation.common.transformers.ToolkitStructure2ViewerStructure.getCell(ToolkitStructure2ViewerStructure.java:850) at com.databasepreservation.common.transformers.ToolkitStructure2ViewerStructure.getCells(ToolkitStructure2ViewerStructure.java:774) at com.databasepreservation.common.transformers.ToolkitStructure2ViewerStructure.getRow(ToolkitStructure2ViewerStructure.java:755) at com.databasepreservation.modules.viewer.DbvtkExportModule.handleDataRow(DbvtkExportModule.java:135) at com.databasepreservation.model.modules.filters.IdentityFilter.handleDataRow(IdentityFilter.java:88) at com.databasepreservation.model.modules.filters.ObservableFilter.handleDataRow(ObservableFilter.java:123) at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.endElement(SIARD2ContentImportStrategy.java:374) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.endElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

[hmiguim]

I need a bit more of context, if possible. And I would like to have the log file if possible.

[Laurira]

I tried to reproduce it but the processess got stuck without any progress this time. Different error at java.lang.Thread.run(Thread.java:748) 2021-09-03 06:32:36,067 [Thread-6] WARN o.a.c.loader.WebappClassLoaderBase - The web application [ROOT] appears to have started a thread named [logback-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

No more log. Should I re-start everything?

[hmiguim]

Yes, please

[Laurira]

Now it started preparing but I think we still have problem here.

For example if user Jack wants to get 3 big databases (A, B, and C) into one dbptk enterprise then I am not sure this can be done. After waited 14.3 hours the database A successfully imported and ready for browsing - Jack is happy and all the users (Mary and Fiona) who need access to the database A can get it using their arhivist accounts. Mary and Fiona are also happy.

Now Jack wants to prepare database B for John but gets "memory leak"-error. Now John is not happy. Jack needs to re-start everything to help John but then database A will be lost from Mary and Fiona. So thats 14.3 hours of wasted time.

I noticed that actually the tmp files are still big in the system but the database is not visible in the DBPTK Enterprise after re-starting everything (docker-compose down/up)

[hmiguim]

The docker-compose provided is for running local tests only. For production environment there are a couple of things that need to be done.

I suggest you take a look at the documentation about docker volumes https://docs.docker.com/storage/volumes/ and https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose