snowstorm
snowstorm copied to clipboard
Published
20 hours ago •
IHTSDO
IHTSDO
Running snowstorm on OpenJDK 11.0.11 fails - (Looks like disk space issue)
Hey everyone, I've been trying to run snowstorm on my Ubuntu LTS 20.04 server. I tried running using both, the docker compose file and running the jar through the command line. However both run into a similar issue and the application does not start. I'm pasting the log below, I hope it helps
2022-08-30 10:51:39.420 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'codeSystemController': Unsatisfied dependency expressed through field 'permissionService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'permissionService': Invocation of init method failed; nested exception is org.springframework.data.elasticsearch.BulkFailureException: Bulk operation has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{global-ADMIN=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [admin-permission] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];]]}]
2022-08-30 10:51:39.427 INFO 1 --- [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Identifier cache manager sleep interrupted.
2022-08-30 10:51:39.428 INFO 1 --- [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Identifier cache manager polling stopped.
2022-08-30 10:51:39.700 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2022-08-30 10:51:39.728 INFO 1 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-08-30 10:51:39.733 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'codeSystemController': Unsatisfied dependency expressed through field 'permissionService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'permissionService': Invocation of init method failed; nested exception is org.springframework.data.elasticsearch.BulkFailureException: Bulk operation has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{global-ADMIN=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [admin-permission] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];]]}]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at org.snomed.snowstorm.SnowstormApplication.main(SnowstormApplication.java:58)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'permissionService': Invocation of init method failed; nested exception is org.springframework.data.elasticsearch.BulkFailureException: Bulk operation has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{global-ADMIN=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [admin-permission] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];]]}]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
... 28 common frames omitted
Caused by: org.springframework.data.elasticsearch.BulkFailureException: Bulk operation has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{global-ADMIN=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [admin-permission] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];]]}]
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.checkForBulkOperationFailure(AbstractElasticsearchTemplate.java:408)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doBulkOperationOnSingleIndex(ElasticsearchRestTemplate.java:244)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doBulkOperation(ElasticsearchRestTemplate.java:252)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.bulkIndex(ElasticsearchRestTemplate.java:198)
at org.springframework.data.elasticsearch.core.DocumentOperations.bulkIndex(DocumentOperations.java:152)
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:175)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.saveAll(AbstractElasticsearchRepository.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.data.repository.core.support.ImplementationInvocationMetadata.invoke(ImplementationInvocationMetadata.java:72)
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:382)
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:205)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:549)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:155)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:130)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy171.saveAll(Unknown Source)
at org.snomed.snowstorm.core.data.services.PermissionService.saveAll(PermissionService.java:193)
at org.snomed.snowstorm.core.data.services.PermissionService.save(PermissionService.java:186)
at org.snomed.snowstorm.core.data.services.PermissionService.setGlobalOrBranchRoleGroups(PermissionService.java:157)
at org.snomed.snowstorm.core.data.services.PermissionService.setGlobalRoleGroups(PermissionService.java:134)
at org.snomed.snowstorm.core.data.services.PermissionService.setGlobalRoleGroups(PermissionService.java:129)
at org.snomed.snowstorm.core.data.services.PermissionService.init(PermissionService.java:49)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
... 40 common frames omitted
2022-08-30 10:51:40.084 INFO 1 --- [-status-polling] o.s.s.c.d.s.c.ClassificationService : Classification status polling thread stopped.
Here's what I get when I do a java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)
Elastic search runs fine, snowstorm seems to be the issue for me.
Hi @TheFlash98, sorry to hear that you are running into trouble and that the error message is not easy to decipher. Thank you for reaching out and including the logs!
This line (last 'Caused by:') tells us the problem:
.... Elasticsearch exception [type=cluster_block_exception, reason=index [admin-permission] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];]]
This tells us that Elasticsearch has locked the index data store. This causes startup to fail because Snowstorm attempts to save a few records during initialisation.
This is usually caused by Elasticsearch not having enough disk space. Please refer to #409 that has details of how to solve this.
I hope that helps! Let us know how it goes.
Kind regards, Kai Kewley
Hey @kaicode thank you so much for the prompt reply. Would have taken me hours before zeroing down on the fact that elastic search was the real issue :smile:
My system had 1TB/15TB free but turns out elastic looks at percentage disk occupied rather than the actual bytes. Simple solution here worked well for me.
Thank you again!
Really glad you fixed it! Thanks for sharing that nicely worded explanation and solution on stack overflow 👍