Restarting degraded Horreum service fails to process messages in dataset-event queue

[johnaohara]

Describe the bug After a Horreum outage, a number of messages were left in the amq dataset-event queue;

After restart, the messages were not processed due to ;

2024-07-09 07:24:41,467 600bc0306043 quarkus-run.jar[7] INFO  [io.sma.rea.mes.pro.con.WorkerPoolRegistry] (vert.x-eventloop-thread-1) Created worker pool named horreum.dataset.pool with concurrency of 7
2024-07-09 07:24:41,769 600bc0306043 quarkus-run.jar[7] INFO  [io.quarkus] (main) horreum-backend 0.14.1 on JVM (powered by Quarkus 3.12.0) started in 5.444s. Listening on: https://0.0.0.0:8443
2024-07-09 07:24:41,770 600bc0306043 quarkus-run.jar[7] INFO  [io.quarkus] (main) Profile prod activated. 
2024-07-09 07:24:41,770 600bc0306043 quarkus-run.jar[7] INFO  [io.quarkus] (main) Installed features: [agroal, cdi, elasticsearch-rest-client, hibernate-orm, hibernate-orm-panache, hibernate-validator, jdbc-postgresql, liquibase, mailer, 
messaging, messaging-amqp, narayana-jta, oidc, quinoa, qute, reactive-pg-client, rest, rest-jackson, resteasy-client, resteasy-client-jackson, resteasy-client-jaxb, scheduler, security, security-jpa, smallrye-context-propagation, smallrye
-openapi, vertx]
2024-07-09 07:24:44,845 600bc0306043 quarkus-run.jar[7] ERROR [io.sma.rea.mes.provider] (horreum.dataset.pool-7) SRMSG00200: The method io.hyperfoil.tools.horreum.svc.ServiceMediator#processDatasetEvents has thrown an exception: jakarta.p
ersistence.OptimisticLockException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [io.hyperfoil.tools.horreum.entity.FingerprintDAO#128366]
        at org.hibernate.internal.ExceptionConverterImpl.wrapStaleStateException(ExceptionConverterImpl.java:209)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:95)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:167)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:173)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1429)
        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1411)
        at org.hibernate.query.sql.internal.NativeQueryImpl.prepareForExecution(NativeQueryImpl.java:601)
        at org.hibernate.query.spi.AbstractSelectionQuery.beforeQuery(AbstractSelectionQuery.java:452)
        at org.hibernate.query.spi.AbstractSelectionQuery.beforeQueryHandlingFetchProfiles(AbstractSelectionQuery.java:439)
        at org.hibernate.query.spi.AbstractQuery.executeUpdate(AbstractQuery.java:648)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl.calculateLabelValues(DatasetServiceImpl.java:452)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_Subclass.calculateLabelValues$$superforward(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_Subclass$$function$$6.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.hyperfoil.tools.horreum.server.RolesInterceptor.intercept(RolesInterceptor.java:70)
        at io.hyperfoil.tools.horreum.server.RolesInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:136)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:107)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:38)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:61)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:32)
       at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_Subclass.calculateLabelValues(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl.onNewDatasetNoLock(DatasetServiceImpl.java:567)
        at io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_ClientProxy.onNewDatasetNoLock(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator.processDatasetEvents(ServiceMediator.java:154)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator_Subclass.processDatasetEvents$$superforward(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator_Subclass$$function$$6.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at io.quarkus.arc.impl.ActivateRequestContextInterceptor.invoke(ActivateRequestContextInterceptor.java:129)
        at io.quarkus.arc.impl.ActivateRequestContextInterceptor.aroundInvoke(ActivateRequestContextInterceptor.java:33)
        at io.quarkus.arc.impl.ActivateRequestContextInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator_Subclass.processDatasetEvents(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator_ClientProxy.processDatasetEvents(Unknown Source)
        at io.hyperfoil.tools.horreum.svc.ServiceMediator_SmallRyeMessagingInvoker_processDatasetEvents_ebe511ee548f91c634de3d182c814a6da94b30b1.invoke(Unknown Source)
        at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$invokeBlocking$15(AbstractMediator.java:190)
        at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromDeferredSupplier.subscribe(UniCreateFromDeferredSupplier.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
        at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)
        at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:88)
        at io.vertx.mutiny.core.WorkerExecutor$1.handle(WorkerExecutor.java:105)
        at io.vertx.mutiny.core.WorkerExecutor$1.handle(WorkerExecutor.java:103)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
        at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1495)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [io.hyperfoil.tools.horreum.entity.FingerprintDAO#128366]
        at org.hibernate.engine.jdbc.mutation.internal.ModelMutationHelper.identifiedResultsCheck(ModelMutationHelper.java:75)
        at org.hibernate.persister.entity.mutation.AbstractDeleteCoordinator.lambda$doStaticDelete$2(AbstractDeleteCoordinator.java:283)
        at org.hibernate.engine.jdbc.mutation.internal.ModelMutationHelper.checkResults(ModelMutationHelper.java:50)
        at org.hibernate.engine.jdbc.mutation.internal.AbstractMutationExecutor.performNonBatchedMutation(AbstractMutationExecutor.java:141)
        at org.hibernate.engine.jdbc.mutation.internal.MutationExecutorSingleNonBatched.performNonBatchedOperations(MutationExecutorSingleNonBatched.java:55)
        at org.hibernate.engine.jdbc.mutation.internal.AbstractMutationExecutor.execute(AbstractMutationExecutor.java:55)
        at org.hibernate.persister.entity.mutation.AbstractDeleteCoordinator.doStaticDelete(AbstractDeleteCoordinator.java:279)
        at org.hibernate.persister.entity.mutation.AbstractDeleteCoordinator.delete(AbstractDeleteCoordinator.java:87)
        at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:131)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:632)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:499)
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:371)
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:41)
        at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:127)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1425)
       ... 61 more

2024-07-09 07:24:44,851 600bc0306043 quarkus-run.jar[7] ERROR [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-1) SRMSG16219: A message sent to channel `dataset-event-in` has been nacked, rejecting the AMQP message and fail-stop