CDCgov
Update FHIRFunctions to not hardcode name of next event
User Story
As a step in the Universal Pipeline,
I want to be able to put QueueMessages on any pipeline step instead of hardcoding the pipeline step that comes after,
so that the Universal Pipeline as a whole can support a less rigid architecture.
Description/Use Case
Presently, a UP function can generate one or more queue messages which bubble up to the FHIRFunctions.do<step> functions, which all follow this pattern:
internal fun doRoute(
message: String,
dequeueCount: Int,
fhirEngine: FHIRRouter,
actionHistory: ActionHistory = ActionHistory(TaskAction.route),
) {
val messagesToDispatch = runFhirEngine(message, dequeueCount, fhirEngine, actionHistory)
messagesToDispatch.forEach {
queueAccess.sendMessage(
elrTranslationQueueName,
it.serialize()
)
}
}
The part of this pattern that is not common is the first param of the sendMessage call, which is a hardcoded queue name. In the case of the route function, every QueueMessage it generates will be sent out to the translation queue. Instead, we should somehow deduce the name of the queue from the concrete type of the QueueMessage.
This ticket was created following the update to the translation function in #13018 where it can now generate a ReportQueueMessage instead of no QueueMessages (the default case for translate is it will upload the blob and update DB and then wait for batchdecider to pick it up. When Topic.isSendOriginal is enabled, we want to skip the batch and directly call send).
Risks/Impacts/Considerations
Dev Notes
Acceptance Criteria
- [ ] FHIRFunctions no longer hardcode which queue the QueueMessage gets sent to
Hey team! Please add your planning poker estimate with Zenhub @arnejduranovic @jack-h-wang @jalbinson @JessicaWNava @JFU-NAVA-PBC @jimmyfagan @mkalish @thetaurean
notes from convo with @arnejduranovic re: clarification of this ticket
davidholiday :monkey: 22 minutes ago I get the ask but there are some specifics I need some clarity on
davidholiday :monkey: 21 minutes ago if we're not "hard coding" message routing from one pipeline segment to the next there are two alternatives that come to mind configuration based routing context based routing - as in - something about the report itself or what happened prior or some other constellation of factors at that time drives the logic determining where the message goes next
davidholiday :monkey: 20 minutes ago I'm assuming y'all are looking for (2) - but let's gain agreement on that. Is that what y'all want?
davidholiday :monkey: 20 minutes ago next question - what's determining where a message goes in terms of business logic ?
davidholiday :monkey: 19 minutes ago that is - assuming I'm right and we're talking about (2) and not (1), what conditions specifically are determining whether or not the "default" behavior of the pipeline is short circuited?
davidholiday :monkey: 18 minutes ago lastly - my read of this is that the ask is NOT to actually short circuit routing so much as creating the means for us to do so in future. is that correct?
davidholiday :monkey: 18 minutes ago ty in advance for clarifying :slightly_smiling_face: (edited)
Arnej Duranović 13 minutes ago Good questions, David! I'll do my best to explain down below and let me know if you have additional comments or questions.
Arnej Duranović 4 minutes ago As the ticket explains, we would like to somehow deduce the name of the queue from the concrete type of the QueueMessage., which I think falls under option TWO you have above. Take a look at doRoute in FHIRFunctions.kt. Each Universal Pipeline specific step follows this pattern and has a similar function in this file. These doX functions are essentially the entry-points for these universal pipeline steps. The first thing they do is call runFhirEngine, which does the appropriate step's business logic. In the case of the route step, this is in FHIRRouter. Notice toward the end of FHIRRouter.doWork we are returning a FHIREngineResult that contains a FhirTranslateQueueMessage! This queue message is what is going to get put up in the azure queue and what is going to trigger the Translate function to run. The goal of this ticket is to use this, or maybe other, information to deduce the first parameter of queueAccess.sendMessage( or maybe we want to eliminate the first param all-together.
Arnej Duranović 4 minutes ago If this doesn't make sense I think it'll be easier to hop o na call and walk through it quick
davidholiday :monkey: 2 minutes ago no I think I get it. basically the result of processing gives us a queuemessage and the type of the message indicates where that message has to go next
davidholiday :monkey: 1 minute ago so the work - fundamentally - is to make changes such that pipeline step work always returns a typed queue message so the last action taken in the current pipeline stage knows where to plop the queue message for further processing (edited)
attempting to add the necessary behavior to the queue message itself. this way the message knows how to route itself. moreover the logic can be short circuited if need be. not sure if this is going to work w/o creating a mess. if it looks like it's going to be a problem I'll eject and do this the naive way by creating a method in FhirFunctions that does what I'm trying to make the queuemessage itself do
I've added behavior to all the queue message types that result in routing to have the message itself self-route. there are two failing tests. I'm not sure yet but I think the tests are at fault but I'm not sure. This is gonna take some time to sort out as all of this is a bit meticulous
of the two tests that were failing, one required a minor refactor as it was endeavoring to put a "route" typed message on the "translation" queue. The other test was failing because an update was required in my logic. PR is up.
more revision requests.
branch became stale to the point where merging master into it broke things
it became cleaner/easier to just redo the whole thing. which I did.
here's the PR
https://github.com/CDCgov/prime-reportstream/pull/15613