CDCgov
Merge data classes in PrimeRouterQueueMessage into QueueMessage in the shared module
User Story
As a RS engineer I want all queue message data classes to inherit from QueueMessage and reside in the shared module, so that the types are not separated and can be shared across the micro-services.
Description/Use Case
Currently all the data classes for the prime-router queue messages inherit from PrimeRouterQueueMessage. These should be moved to QueueMessage in the shared module to simplify the code.
Dev Notes
This will include moving additional classes out of prime-router or determining which classes are no longer needed. Currently these are used in PrimeRouterQueueMessage:
import gov.cdc.prime.router.Options
import gov.cdc.prime.router.ReportId
import gov.cdc.prime.router.Topic
import gov.cdc.prime.router.azure.Event
import gov.cdc.prime.router.azure.QueueAccess
A review of the fields included in the queue message classes is important including the purpose of those fields in the pipeline steps. Review which fields are necessary and of the necessary fields, can the data be acquired outside of the queue message (e.g. via a DB query).
Acceptance Criteria
- [ ] PrimeRouterQueueMessage no longer holds any queue messages classes for UP steps
- [ ] Ensure build dependencies are properly scoped to limit leakage
Hey team! Please add your planning poker estimate with Zenhub @adegolier @arnejduranovic @brick-green @david-navapbc @jack-h-wang @jalbinson @JFisk42 @mkalish @thetaurean
Been noodling with moving parts of the code into its own module and realized that in the course of pulling classes out there will be cases where a dependency is required by both what we pulled out and by other classes that remain in the monolith. I settled on a creating a version catalogue as part of our gradle build. This way, in cases where we need a dependency to live in multiple places, we can manage the lifecycle of every instance of that dependency in every project using the catalogue to include said dependency in one central place.
I tested the approach using the :submissions microservice spring-boot plugin definition. The new auth microservice also uses spring boot and thus we can use the catalogue to manage the spring-boot version for both subprojects in once place.
Next step is to go back to square one with migration of classes and start moving things again, but this time, for every class migrated, also update the build files for both the monolith and the sub project such that any dependencies that class touches are:
- migrated to the new subproject if the sub project is the only thing that requires it
- kept in both but managed by the version catalogue
I tried whole hog ripping things out of prime but there's too much that's tied in with too much for that to be a tenable solution. I've pivoted to a strategy where I'm pulling out the bare minimum and splitting things like Event into two so the generic stuff can life in the shared lib and the detail stuff can continue to live in prime.
I've successfully migrated the queue message stuff out of PrimeRouterQueueMessage and into the shared module. Additionally I've migrated - in as minimal a manner as possible - some other things like Topic and EventAction into the shared module. I updated all the non-test code references and the jooq bindings for Topic.
What's left is to update the test code to point to what's now in the shared class and to ensure all tests pass.
found and remediated an issue with jackson de-serialization of implementers of iTopic. The remaining issue is same but with the two MARS* Topics that inhereit from iTopic -> iTopicWithValidator. Unfortunately I think I'm going to have to migrate those over to the shared library as well. Hopefully it's not tied into too much else ...
down to three failing tests
gov.cdc.prime.router.fhirengine.engine.FhirConverterTests > FhirConverterProcessTest should log a FHIR validation error and not return a bundle() FAILED
org.opentest4j.AssertionFailedError: expected to be null but was:<org.hl7.fhir.r4.model.Bundle@4c37ce90>
at app//gov.cdc.prime.router.fhirengine.engine.FhirConverterTests$FhirConverterProcessTest.should log a FHIR validation error and not return a bundle(FhirConverterTests.kt:654)
gov.cdc.prime.router.fhirengine.engine.FhirConverterTests > FhirConverterProcessTest should log a HL7 validation error and not return a bundle() FAILED
org.opentest4j.AssertionFailedError: expected to be null but was:<org.hl7.fhir.r4.model.Bundle@76924859>
at app//gov.cdc.prime.router.fhirengine.engine.FhirConverterTests$FhirConverterProcessTest.should log a HL7 validation error and not return a bundle(FhirConverterTests.kt:708)
gov.cdc.prime.router.fhirengine.azure.FHIRConverterIntegrationTests should successfully convert messages for a topic with validation() FAILED (9.4s)
java.lang.IllegalArgumentException: List has more than one element.
at gov.cdc.prime.router.fhirengine.azure.FHIRConverterIntegrationTests.should successfully convert messages for a topic with validation(FHIRConverterIntegrationTests.kt:584)
all failing the same reason
java.lang.IllegalStateException: Unable to load lookup table 'observation-mapping' for code to condition lookup
at gov.cdc.prime.router.azure.LookupTableConditionMapper.<init>(ConditionMapper.kt:19)
at gov.cdc.prime.router.fhirengine.engine.FHIRConverter.process$prime_router(FHIRConverter.kt:408)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.mockk.proxy.jvm.advice.MethodCall.call(MethodCall.kt:14)
at io.mockk.proxy.jvm.advice.SelfCallEliminatorCallable.call(SelfCallEliminatorCallable.kt:14)
at io.mockk.impl.instantiation.JvmMockFactoryHelper.handleOriginalCall(JvmMockFactoryHelper.kt:96)
at io.mockk.impl.instantiation.JvmMockFactoryHelper.access$handleOriginalCall(JvmMockFactoryHelper.kt:19)
at io.mockk.impl.instantiation.JvmMockFactoryHelper$mockHandler$1$invocation$1$1.invoke(JvmMockFactoryHelper.kt:28)
at io.mockk.impl.stub.MockKStub$handleInvocation$originalPlusToString$1.invoke(MockKStub.kt:235)
at io.mockk.impl.stub.SpyKStub.defaultAnswer(SpyKStub.kt:15)
at io.mockk.impl.stub.MockKStub.answer(MockKStub.kt:44)
at io.mockk.impl.recording.states.AnsweringState.call(AnsweringState.kt:16)
at io.mockk.impl.recording.CommonCallRecorder.call(CommonCallRecorder.kt:53)
at io.mockk.impl.stub.MockKStub.handleInvocation(MockKStub.kt:271)
at io.mockk.impl.instantiation.JvmMockFactoryHelper$mockHandler$1.invocation(JvmMockFactoryHelper.kt:24)
at io.mockk.proxy.jvm.advice.Interceptor.call(Interceptor.kt:21)
at gov.cdc.prime.router.fhirengine.engine.FHIRConverter.process$prime_router(FHIRConverter.kt:366)
at gov.cdc.prime.router.fhirengine.engine.FHIRConverter.process$prime_router$default(FHIRConverter.kt:357)
at gov.cdc.prime.router.fhirengine.engine.FhirConverterTests$FhirConverterProcessTest.should handle a parse failure for the entire HL7 batch(FhirConverterTests.kt:589)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
down to one failing test
gov.cdc.prime.router.fhirengine.azure.FHIRConverterIntegrationTests should successfully convert messages for a topic with validation() FAILED (9.8s)
org.opentest4j.AssertionFailedError: expected:<{ITEM_FORMAT=HL7, [VALIDATION_PROFILE="RADx MARS", ]PROCESSING_ERROR="It...> but was:<{ITEM_FORMAT=HL7, []PROCESSING_ERROR="It...>
at app//gov.cdc.prime.router.fhirengine.azure.FHIRConverterIntegrationTests.should successfully convert messages for a topic with validation(FHIRConverterIntegrationTests.kt:584)
all tests passing
pushed additional change to build files that de-duplicates definitions for dependencies that exist in both projects
the requirements on this have changed. I'm looking through the other related tickets and trying to figure out what I need to do to close this out.
going to chat with Arnej again about refactoring submissions so it does not need to put a queue message onto the queue (and thus does not need any of of the dependencies tied to QueueMessage).
I think the best way is to use a BlobStore trigger.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob?tabs=isolated-process%2Cextensionv5%2Cextensionv3&pivots=programming-language-java
submissions service already stores submissions to blob store. That event can trigger prime to process the data.
triggers have configurable retry policies that can be employed so we ensure data gets processed even if prime goes down or there is some other issue in processing the data
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-error-pages?tabs=fixed-delay%2Cisolated-process%2Cnode-v4%2Cpython-v2&pivots=programming-language-java
last but not least - if we for some reason don't want to use the blob trigger we can employ a timer trigger to regularly poll for new input on a time-based cadence
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=python-v2%2Cisolated-process%2Cnodejs-v4&pivots=programming-language-java
Refined this ticket more in https://app.zenhub.com/workspaces/platform-6182b02547c1130010f459db/issues/gh/cdcgov/prime-reportstream/16239. Will close this one as duplicate.
