DataDictionary necessary for parsing repeating groups...

[cbusbey]

For the most part, generated messages and fields suffice for sending and receiving FIX messages with repeating groups. However... it is not possible for session code unaware of the message structure to parse a FIX message with repeating groups, and send that message on a FIX session.

This very scenario plays out with resends. The sender sends a flattened FIX message, the bytes of that FIX message are persisted in the store. On a resend request, the session code must parse these bytes, allow ToApp/ToAdmin to modify the the message, and resend to the target. The current code does not use a data dictionary on parse. As a result, the resent message does not resemble original message with repeating groups.

Note that all the other quickfix impls also require a datadictionary for parsing repeating groups, though this will incur latency hits.

[imirkin]

We're using this patch for resends specifically, which seems to work OK. Obviously a hack, esp since it will end up ignoring anything that ToApp does to the body, but ... better than broken messages. (And I don't think it's particularly common to change the body for resends.)

diff --git a/vendor/github.com/quickfixgo/quickfix/in_session.go b/vendor/github.com/quickfixgo/quickfix/in_session.go
index d4dfc701d..b9c9460c5 100644
--- a/vendor/github.com/quickfixgo/quickfix/in_session.go
+++ b/vendor/github.com/quickfixgo/quickfix/in_session.go
@@ -238,7 +238,7 @@ func (state inSession) resendMessages(session *session, beginSeqNo, endSeqNo int
                }
 
                session.log.OnEventf("Resending Message: %v", sentMessageSeqNum)
-               msgBytes = msg.build()
+               msgBytes = msg.resendBuild()
                session.sendBytes(msgBytes)
 
                seqNum = sentMessageSeqNum + 1
diff --git a/vendor/github.com/quickfixgo/quickfix/message.go b/vendor/github.com/quickfixgo/quickfix/message.go
index f800cbebf..345ec1be9 100644
--- a/vendor/github.com/quickfixgo/quickfix/message.go
+++ b/vendor/github.com/quickfixgo/quickfix/message.go
@@ -373,9 +373,32 @@ func (m *Message) build() []byte {
        return b.Bytes()
 }
 
+func (m *Message) resendBuild() []byte {
+       m.resendCook()
+
+       var b bytes.Buffer
+       m.Header.write(&b)
+       b.Write(m.bodyBytes)
+       m.Trailer.write(&b)
+       return b.Bytes()
+}
+
 func (m *Message) cook() {
        bodyLength := m.Header.length() + m.Body.length() + m.Trailer.length()
        m.Header.SetInt(tagBodyLength, bodyLength)
        checkSum := (m.Header.total() + m.Body.total() + m.Trailer.total()) % 256
        m.Trailer.SetString(tagCheckSum, formatCheckSum(checkSum))
 }
+
+func (m *Message) resendCook() {
+       bodyLength := m.Header.length() + len(m.bodyBytes) + m.Trailer.length()
+       m.Header.SetInt(tagBodyLength, bodyLength)
+
+       bodyTotal := 0
+       for _, b := range []byte(m.bodyBytes) {
+               bodyTotal += int(b)
+       }
+
+       checkSum := (m.Header.total() + bodyTotal + m.Trailer.total()) % 256
+       m.Trailer.SetString(tagCheckSum, formatCheckSum(checkSum))
+}

[imirkin]

@cbusbey would it be worthwhile to get the above into the quickfixgo codebase, or should we keep it in our vendor tree?

[cbusbey]

Thanks @imirkin but I'm hoping we get a more complete fix in shortly, I'm aware of some work being done on this now. Not sure on the ETA though, your snippet here might be useful to others in the interim.

[ackleymi]

I'm working up a fix https://github.com/quickfixgo/quickfix/compare/master...ackleymi:master that uses a datadictionary to aid in parsing repeating groups. It's still a WIP with a few concerns to iron out, but might serve as a step in the right direction.

[adampinky85]

Hi all, is there any update on this issue? We just debugged the same issue where a resend request resulted in malformed response due to missing repeating groups. The message is subsequently rejected.

[imirkin]

@cbusbey It's now been ~1.5 years since I posted the hack-fix. It's a hack, but an effective one. Nothing better has materialized since then. Making resends work seems like an important use-case. Let's not let perfect be the enemy of good?

[cbusbey]

Hi @imirkin

I've stepped away from this project, but perhaps @ackleymi can give you an update?

[steelkorbin]

@cbusbey thank you for all that you have done over the years. @ackleym are you going to taking the helm on this project?

[ackleymi]

@steelkorbin correct, and happy to build on top of chris' work. @imirkin I can work this week to get the hack merged in the meantime, and I've recently resumed some work on a more comprehensive fix based on Qfix-J's approach.

[ackleymi]

Agreed that repeating group resends is a very common must-have

[totalys]

Anyone working in this issue right now? This topic seems old and this is a must-have... If it is not being handled I guess I'll give it a try... is there any started work on this ?

[totalys]

or maybe just add the hack from @imirkin ... I guess it would work for me...

[nbalsamo]

Hi @totalys @imirkin @ackleymi any way I can help get this through? We're running into this issue as well.

@imirkin did you create a branch with your change?

[imirkin]

@imirkin did you create a branch with your change?

No ... I thought @ackleymi said he'd do it properly... If someone says they'll apply it, then I can get a PR created with this hacky change, otherwise it's just busy-work. Note the hack change does nothing to address the original parsing issue, it only makes resends work.

[acronin-stash]

Hey @ackleymi I see you are assigned this issue. Do you have any updates on the status? We have also run into this problem and are trying to determine best path forward.

[totalys]

@ackleymi , we need this too.