hapifhir
v6.6.0+ Docker images all failing to start in R5 mode
The "latest" Docker Hub repository tag (as of today), v6.8.0, and v6.6.0 docker images are all failing to start in R5 mode with a Spring runtime exception at startup. It appears to be a missing IEmailSender bean definition. This seems to have been introduced after v6.4.4, as v6.4.4 starts as expected. For example, docker run -it --rm -p 8080:8080 -e hapi.fhir.fhir_version=R5 hapiproject/hapi:latest (and v6.8.0 and v6.6.0) always fail to start with the following runtime exception:
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'subscriptionDeliveryHandlerFactory' defined in ca.uhn.fhir.jpa.subscription.match.config.SubscriptionProcessorConfig: Unsatisfied dependency expressed through method 'subscriptionDeliveryHandlerFactory' parameter 1; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'ca.uhn.fhir.jpa.subscription.match.deliver.email.IEmailSender' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
... 56 common frames omitted
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'ca.uhn.fhir.jpa.subscription.match.deliver.email.IEmailSender' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1801)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1396)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
... 69 common frames omitted
Also reported via Zulip, here: https://chat.fhir.org/#narrow/stream/179167-hapi/topic/R5.20Docker.20Crashing/near/395149283
Hi @jkiddo, Thank you for working on this! Any chance we could get a new build? I'm happy to test it. :) Unfortunately this bug means that there are no builds supporting R5 final, as 6.4.4 implements 5.0.0-ballot and has some differences. (The *-tomcat builds always starts in R4 mode and seem ignore the hapi.fhir.fhir_version variable.)
Understood.. Who can validate and authorize this? I'd like to politely request a patch release.
I just Approved @jkiddo 's PR now. @preston please confirm that everything looks good to you now, thanks!
Thank you @KevinDougan-SmileCDR ! Do the "latest" Docker Hub images get built automatically by CI? ..or is that a separate request?
No ... latest only reflects latest released right now. Also, this patch goes to 6.8.x. @dotasek would you be able to do a patch release if @preston can verify it works for him locally?
@preston you should be able to build it locally only using docker from the Dockerfile in the repo.
I definitely could be building it wrong, but on a Ubuntu x64 machine:
docker build -t p3000/hapi-fhir-jpaserver-starter:latest .
docker run -it --rm -p 8080:8080 -e hapi.fhir.fhir_version=R5 p3000/hapi-fhir-jpaserver-starter:latest
Throws the following JDBC-related exception at startup.
Caused by: org.hibernate.exception.GenericJDBCException: Unable to open JDBC Connection for DDL execution
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.transaction.backend.jdbc.internal.DdlTransactionIsolatorNonJtaImpl.getIsolatedConnection(DdlTransactionIsolatorNonJtaImpl.java:71)
at org.hibernate.tool.schema.internal.exec.ImprovedExtractionContextImpl.getJdbcConnection(ImprovedExtractionContextImpl.java:63)
at org.hibernate.tool.schema.extract.spi.ExtractionContext.getQueryResults(ExtractionContext.java:43)
at org.hibernate.tool.schema.extract.internal.SequenceInformationExtractorLegacyImpl.extractMetadata(SequenceInformationExtractorLegacyImpl.java:39)
at org.hibernate.tool.schema.extract.internal.DatabaseInformationImpl.initializeSequences(DatabaseInformationImpl.java:66)
at org.hibernate.tool.schema.extract.internal.DatabaseInformationImpl.<init>(DatabaseInformationImpl.java:60)
at org.hibernate.tool.schema.internal.Helper.buildDatabaseInformation(Helper.java:183)
at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.doMigration(AbstractSchemaMigrator.java:104)
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:196)
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:85)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:335)
at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:471)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1498)
at org.hibernate.jpa.HibernatePersistenceProvider.createContainerEntityManagerFactory(HibernatePersistenceProvider.java:141)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:409)
... 107 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: Error while creating file "/app/target" [90062-220]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:566)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
at org.h2.message.DbException.get(DbException.java:212)
at org.h2.store.fs.disk.FilePathDisk.createDirectory(FilePathDisk.java:391)
at org.h2.store.fs.FileUtils.createDirectory(FileUtils.java:85)
at org.h2.store.fs.FileUtils.createDirectories(FileUtils.java:382)
at org.h2.store.fs.FileUtils.createDirectories(FileUtils.java:381)
at org.h2.mvstore.db.Store.<init>(Store.java:106)
at org.h2.engine.Database.<init>(Database.java:324)
at org.h2.engine.Engine.openSession(Engine.java:92)
at org.h2.engine.Engine.openSession(Engine.java:222)
at org.h2.engine.Engine.createSession(Engine.java:201)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:338)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:122)
at org.h2.Driver.connect(Driver.java:59)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:100)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:181)
at org.hibernate.resource.transaction.backend.jdbc.internal.DdlTransactionIsolatorNonJtaImpl.getIsolatedConnection(DdlTransactionIsolatorNonJtaImpl.java:44)
... 122 common frames omitted
Caused by: java.nio.file.AccessDeniedException: /app/target
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
at java.base/java.nio.file.Files.createDirectory(Files.java:700)
at org.h2.store.fs.disk.FilePathDisk.createDirectory(FilePathDisk.java:382)
... 143 common frames omitted
The Dockerfile creates a /target dir, but not /app/target. Looks like the nonroot user doesn't have permissions to create it. Probably an easy fix, but I'm not sure how it was working previously.
@jkiddo after reflecting on this, I think this should be image/v6.8.0.1, if we're going off the master branch. We're bound to the HAPI parent pom, which is still 6.8.0, so I don't want to create confusion.
But technically the starter project 'just' consumes the base project so the versioning doesnt have to be aligned - but I follow you
@jkiddo Nope. Clean checkout of master. Build and run with the exact commands and throws the runtime exception about /app/target every time. Docker version 24.0.5, build 24.0.5-0ubuntu1~23.04.1
The Dockerfile creates a /target dir, but not /app/target. Looks like the nonroot user doesn't have permissions to create it. Probably an easy fix, but I'm not sure how it was working previously.
@XcrigX where does it create a /target dir in the Dockerfile for the default build? I can see its the case for the tomcat one but not for the distroless (default) variant
@preston can you verify that the rest of it works by running docker run -it --rm -p 8080:8080 -e hapi.fhir.fhir_version=R5 -e spring.datasource.url=jdbc:h2:mem:test_mem p3000/hapi-fhir-jpaserver-starter:latest
The Dockerfile creates a /target dir, but not /app/target. Looks like the nonroot user doesn't have permissions to create it. Probably an easy fix, but I'm not sure how it was working previously.
@XcrigX where does it create a /target dir in the Dockerfile for the default build? I can see its the case for the tomcat one but not for the distroless (default) variant
@jkiddo - I see.. I wasn't looking that closely. Regardless, it seems that directory needs to pre-exist? But I may just be misunderstanding something because again, I'm not clear how it was working previously without that? Perhaps ignore me if I'm not being helpful here. ;)
@XcrigX not at all - you're on to something here, but this part has not been changed for ages. Biggest problem is that I cannot reproduce it locally.
@preston can you verify that the rest of it works by running
docker run -it --rm -p 8080:8080 -e hapi.fhir.fhir_version=R5 -e spring.datasource.url=jdbc:h2:mem:test_mem p3000/hapi-fhir-jpaserver-starter:latest
@jkiddo Similar runtime exception having to do with /app/target :
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: HAPI_PU] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:
index 'TermConcept':
failures:
- HSEARCH600154: Unable to start index: HSEARCH600015: Unable to initialize index directory: /app/target
Context: index 'TermConcept'
index 'ResourceTable':
failures:
- HSEARCH600154: Unable to start index: HSEARCH600015: Unable to initialize index directory: /app/target
Context: index 'ResourceTable'
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:421)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:396)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
... 103 common frames omitted
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:
index 'TermConcept':
failures:
- HSEARCH600154: Unable to start index: HSEARCH600015: Unable to initialize index directory: /app/target
Context: index 'TermConcept'
index 'ResourceTable':
failures:
- HSEARCH600154: Unable to start index: HSEARCH600015: Unable to initialize index directory: /app/target
Context: index 'ResourceTable'
at org.hibernate.search.engine.reporting.spi.RootFailureCollector.checkNoFailure(RootFailureCollector.java:53)
at org.hibernate.search.engine.common.impl.SearchIntegrationPartialBuildStateImpl$SearchIntegrationFinalizerImpl.finalizeIntegration(SearchIntegrationPartialBuildStateImpl.java:188)
at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationPartialBuildState.doBootSecondPhase(HibernateOrmIntegrationPartialBuildState.java:75)
at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.bootNow(HibernateOrmIntegrationBooterImpl.java:176)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateSearchSessionFactoryObserver.sessionFactoryCreated(HibernateSearchSessionFactoryObserver.java:41)
at org.hibernate.internal.SessionFactoryObserverChain.sessionFactoryCreated(SessionFactoryObserverChain.java:35)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:402)
at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:471)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1498)
at org.hibernate.jpa.HibernatePersistenceProvider.createContainerEntityManagerFactory(HibernatePersistenceProvider.java:141)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:409)
... 107 common frames omitted
FWIW, I get the same error when running @preston 's build/run commands locally. Fresh clone of master - I'm running in a WSL2 (windows) Ubuntu 20.x instance, not that I think it would make any difference.
In lieu of a patch release, I did release jpa-s-s 6.8.3. This was an opportunity to update to HAPI 6.8.3. If another patch release is required, please ping me here, or on zulip.
Thanks all for working on this! The 6.8.3 release seems to be working now as expected when started in R5 mode.
The CI pipeline builds definitely are not build by a root user, hence it makes sense that the 6.8.3 works. @preston the local build you did - was that done using a regular user i a folder not owned by anyone special?
was that done using a regular user i a folder not owned by anyone special?
@jkiddo Yup .. just a regular Ubuntu user account. Nothing special. I don't think I'm understanding the way the distroless builds work regarding user/group filesystem permissions.
The build environment/builder doesn't matter, does it? All those permissions/directories are within the docker build/container, no?
It might be something related to the way different builder and runtime implementations are configured to map permissions to a host systems both at build time and run time, which is somewhat configurable per installation. I'm guessing everyone here is using Docker Desktop for local use, but I recall having issues with other projects in the past with images assuming root vs non-root permissions at runtime.
@dotasek What CI system and builder builds these? They are multiarch (thank you for supporting that too btw) and we are also using the ARM64 images.
@preston I believe these are all built via this GitHub action: https://github.com/hapifhir/hapi-fhir-jpaserver-starter/blob/master/.github/workflows/build-images.yaml
This is my first time taking a look at that file, so this is just an educated guess based on the action trigger and the tagging I performed for the last two releases.
They'll run on whatever ubuntu-20.04 GitHub runners are available. Any info regarding the docker steps and permissions is likely in buried in GitHub actions documentation.