Configuration parameter `ValidateFieldsOutOfOrder` not working when header fields found in body part

[jacques0803]

Discussed in https://github.com/quickfix-j/quickfixj/discussions/467

^{Originally posted by jacques0803 February 14, 2022} Hi FIX Community,

I'm attempting to exchange messages with a party that does properly order fields. When ValidateFieldsOutOfOrder=N, simpler messages such as Logon and TradeExecutionReportRequestAck message works fine, as all message header fields are at the start of the message, albeit not in order.

The problem I'm facing now is that the TradeExecutionReport fails to process (Rejected with "Received message without MsgSeqNum") as TAG34 is present in the message body, and not the header.

ValidateFieldsOutOfOrder: If set to N, fields that are out of order (i.e. body fields in the header, or header fields in the body) will not be rejected. Useful for connecting to systems which do not properly order fields.

So, my assumption is that, even if the Header field is provided within the message body, QuickFix/J should still be able to extract the relevant fields and populate the Message.Header object correctly.

QuickFix/J Version 2.3.1 JDK 1.8 FIX5.0SP2

TradeExecutionReport Test Message 1 - All Header fields at start of message, and reading the message works perfect:

8=FIXT.1.1|9=504|35=AE|49=ASX|56=MSA01_3383_TEST|34=545|52=20220210-02:44:00.820|1128=9|487=0|1125=|1003=1120000338|75=20220210|1015=0|64=20220214|60=20220210-02:43:27.796|55=ANZ|48=AU000000ANZ3|22=4|461=Exxxxx|381=135000|31=27|32=5000.000000000000|15=AUD|1301=XASX|20003=XT|20007=XD|552=2|54=1|11=7533509260093758035|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|54=2|11=7533509260093757876|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|880=7533509260093686098:0#NORMAL#1644451200000000000|167=CS|762=1|106=4075|10=087|

TradeExecutionReport Test Message 2 - Only TAG8,9 and 35 at start of message, all other header tags mixed into body of message:

8=FIXT.1.1|9=504|35=AE|487=0|1125=|1003=1120000338|75=20220210|1015=0|64=20220214|60=20220210-02:43:27.796|55=ANZ|48=AU000000ANZ3|22=4|461=Exxxxx|381=135000|31=27|32=5000.000000000000|15=AUD|1301=XASX|20003=XT|20007=XD|552=2|54=1|11=7533509260093758035|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|54=2|11=7533509260093757876|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|49=ASX|56=MSA01_3383_TEST|34=545|52=20220210-02:44:00.820|1128=9|880=7533509260093686098:0#NORMAL#1644451200000000000|167=CS|762=1|106=4075|10=087|

Configuration values I do set to prevent certain validations: ValidateUserDefinedFields: N ValidateFieldsOutOfOrder: N ValidateFieldsHaveValues: N ValidateUnorderedGroupFields: N AllowUnknownMsgFields: Y

The test I perform (test Messages loaded from a resource file):

  public void testSetup() {
      quickfix.fix50sp2.MessageFactory fix50SP2MessageFactory = new quickfix.fix50sp2.MessageFactory();
      ApplVerID applVerID = new ApplVerID(ApplVerID.FIX50SP2);
      DataDictionary applicationDD = new DefaultDataDictionaryProvider(true).getApplicationDataDictionary(applVerID);
      applicationDD.setCheckFieldsOutOfOrder(false);
      applicationDD.setCheckUnorderedGroupFields(false);
      applicationDD.setCheckFieldsHaveValues(false);
      applicationDD.setAllowUnknownMessageFields(true);
      applicationDD.setCheckUserDefinedFields(false);

      final String beginString = "FIXT.1.1";

      new BufferedReader(new InputStreamReader(resource.getInputStream())).lines().forEach(msgString -> {
          TradeCaptureReport message = (TradeCaptureReport) fix50SP2MessageFactory.create(beginString, applVerID, MsgType.TRADE_CAPTURE_REPORT);
          try {
              message.fromString(msgString, applicationDD, true);
              messages.add(message);
          } catch (InvalidMessage e) {
              e.printStackTrace();
          }

      });
  }
  public void testMessages() {
      messages.forEach(message -> {
          if (!Objects.isNull(message.getException())) {
              log.debug("Field Exception {}", message.getException());
          }
          // print Header Fields
          printField(message.getHeader(), 8);
          printField(message.getHeader(), 9);
          printField(message.getHeader(), 35);
          printField(message.getHeader(), 49);
          printField(message.getHeader(), 56);
          printField(message.getHeader(), 34);
          printField(message.getHeader(), 52);
          printField(message.getHeader(), 1128);
      });
  }
  private void printField(quickfix.Message.Header header, int fieldNo) {
      try {
          log.debug("TAG: {}={}", fieldNo, header.getString(fieldNo));
      } catch (FieldNotFound e) {
          log.error("TAG {} not found", fieldNo);
      }
  }

The output of running the test:

-------- Testing message -------- TAG: 8=FIXT.1.1 TAG: 9=504 TAG: 35=AE TAG: 49=ASX TAG: 56=MSA01_3383_TEST TAG: 34=545 TAG: 52=20220210-02:44:00.820 TAG: 1128=9 -------- Testing message -------- TAG: 8=FIXT.1.1 TAG: 9=504 TAG: 35=AE TAG 49 not found TAG 56 not found TAG 34 not found TAG 52 not found TAG 1128 not found

Any help/comments would be greatly appreciated.

Kind regards

[jacques0803]

@chrjohn, I would really appreciate your expertise. Maybe I'm missing a configuration, but whatever I try, I'm unable to to receive a message when Header fields are mixed with body fields.

[chrjohn]

@jacques0803 well, I think there needs to be at least some ordering. There are places in the code that try to access header fields, e.g. for fields that are needed for session identification. This is done prior to validation. QFJ needs at least some info to know which session the message belongs to. Maybe the documentation needs to be clearer on this. But IMHO one cannot expect that fields are in arbitrary places all over the message. Sorry, but I think there is no option that does this.

[jacques0803]

@chrjohn Thank you for your comments.

I agree 100% with you. Header Tags at the front, followed by Body tags and then the Checksum at the end of a message. ANd obviously, order becomes very important when you have repeating groups in the message.

However, I have the following observations on how QFJ is behaving:

1. Even though SenderCompID and TargetCompID is somewhere near the end of the message string, QFJ is still identifying the session correctly and reporting on the missing TAG34. It then responds by issuing a Logout, followed by a Logon message with the NextExpectedMsgSeqNum set to the last known valid message. Is it possible that QFJ is perhaps applying some other logic to search for these session identifying TAGs, and then only parses/validates the actual message?
2. I did some further testing, and if I move these header tags forward on the message string, just in front of the repeating group TrdCapRptSideGrp (TAG552), then QFJ is able to parse the message correctly. So Header tags are still mixed with body tags, and QFJ is able to parse the message correctly: 8=FIXT.1.1|9=419|35=AE|487=0|1125=|1003=1010004257|75=20220301|1015=0|64=20220303|60=20220228-23:15:24.300|55=BHP|48=AU000000BHP4|22=4|461=Exxxxx|381=480|31=48|32=10.000000000000|15=AUD|1301=XASX|20003=|20007=XD|49=ASX|56=MSA01_3383_TEST|52=20220228-23:28:56.477|1128=9|34=4|552=1|54=1|11=7539050794402809736|1=WFC69247S|576=1|577=0|453=1|448=338-3|447=D|452=1|880=7541030190210617505:0#NORMAL#1646092800000000000|167=CS|762=1|106=4172|10=180|

We are having conversations with the counterparty with the hope that they will order the tags so that we can progress development.

[chrjohn]

@jacques0803 there was a comment from you yesterday but it is gone now?!

[jacques0803]

Hi @chrjohn

Apologies for deleting the comment, as I had to be 100% sure of my test scenario and outcome. I did did some more testing/digging and I'm now ready to list my observation/ask question:

Referring to the the message I listed at the top of this conversation, and repeated here:

8=FIXT.1.1|9=504|35=AE|487=0|1125=|1003=1120000338|75=20220210|1015=0|64=20220214|60=20220210-02:43:27.796|55=ANZ|48=AU000000ANZ3|22=4|461=Exxxxx|381=135000|31=27|32=5000.000000000000|15=AUD|1301=XASX|20003=XT|20007=XD|552=2|54=1|11=7533509260093758035|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|54=2|11=7533509260093757876|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|49=ASX|56=MSA01_3383_TEST|34=545|52=20220210-02:44:00.820|1128=9|880=7533509260093686098:0#NORMAL#1644451200000000000|167=CS|762=1|106=4075|10=087|

My understanding of the message parsing flow when a message is received: The AbstractIoHandler.messageReceived passes to the MessageUtils.parse method the Session and the raw string received. In turn, the MessageUtils.parse determines the session and application data dictionaries for the QFJ Session, and then invokes Message.parse with this information. In my case, and cosidering the input message, the session dictionary is FIXT.1.1 and the application dictionary is FIX.5.0SP2.

I've configured the FIX Session with the following parameters (the full configuration at the end of this post):

ValidateUserDefinedFields: N ValidateFieldsOutOfOrder: N ValidateFieldsHaveValues: N ValidateUnorderedGroupFields: N

I can see that these configuration parameters have taken effect on the sessionDataDictionary, as I'm outputting the following log messages in a hacked version of Message.parse method:

sessionDataDictionary.isValidateUserDefinedFields() => false sessionDataDictionary.isValidateFieldsOutOfOrder() => false sessionDataDictionary.isValidateFieldsHaveValues() => false sessionDataDictionary.isValidateUnorderedGroupFields() => false

However, for the applicationDataDictionary argument to the same Message.parse method, the following is returned isValidateUserDefinedFields() => true isValidateFieldsOutOfOrder() => true isValidateFieldsHaveValues() => true isValidateUnorderedGroupFields() => true

To me it almost seems the applicationDataDictionary does not have the originally configured settings (at least not the 4 that I've output), and hence, when it comes to parsing the body of the message, the applicationDataDictionary does not allow for Unordered Group Fields that are present in the test message I listed above.

Am I missing a configuration parameter that will ensure the unordered group fields will not be rejected? For FIX Versions pre-FIXT.1.1, the application and session data dictionaries are the same, and I suspect no issues in configuring the FIX Session to allow for the Unordered Group Fields. When I hack and force the 4 settings onto the application Data Dictionary at the Message.parse method, then repeating groups with the unordered group fields are not rejected with:

throw new FieldException(SessionRejectReason.REPEATING_GROUP_FIELDS_OUT_OF_ORDER, tag);

Herewith my full configuration of the FIX Session:

[SESSION] SenderCompID: the sender TargetCompID: the target ConnectionType: initiator SocketConnectHost: the IP address SocketConnectPort: the port SocketUseSSL: Y SocketKeyStore: /etc/pki/java/cacerts KeyStoreType: JKS SocketTrustStore: /etc/pki/java/cacerts TrustStoreType: JKS FileStorePath: /var/hostlinks/ebbos_var/logs/feed/signalb_fix/dev1 StartTime: "07:00:00" EndTime: "20:30:00" TimeZone: Australia/NSW HeartBtInt: 30 ReconnectInterval: 30 BeginString: FIXT.1.1 DefaultApplVerID: FIX.5.0SP2 LogonTimeout: 600 FileIncludeTimeStampForMessages: Y RejectInvalidMessage: N EnableNextExpectedMsgSeqNum: Y ValidateSequenceNumbers: Y ValidateUserDefinedFields: N ValidateFieldsOutOfOrder: N ValidateIncomingMessage: N ValidateFieldsHaveValues: N ValidateUnorderedGroupFields: N SLF4JLogPrependSessionID: N SLF4JLogIncomingMessageCategory: signalb.msg.incoming SLF4JLogOutgoingMessageCategory: signalb.msg.outgoing

As always, thank you for your time and valuable feedback. Regards @jacques0803

[chrjohn]

Hi @jacques0803

having a brief glance at https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/DefaultSessionFactory.java#L300 it looks like the application dictionaries are created with the same settings via https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/DefaultSessionFactory.java#L316

So to me it looks like the app dictionary should be created with the same settings as the session dictionary. I can't really tell why it is not working for you.

BTW, you did not quote your settings for the data dictionaries. I can see no AppDataDictionary or TransportDataDictionary entries...

[jacques0803]

Hi @chrjohn Seems that the AppDataDictionary configuration parameter is actually required, and I was relying on the default behaviour quoted for the configuration parameter:

If no dictionary path is supplied, an attempt will be made to load a dictionary using the DefaultApplVerID for the session.

I suspect, the Application DataDictionary is not created as part of creating the session because the AppDataDictionary configuration setting is not supplied, and the creation of the App DataDictionary maybe be deferred to a later point in the flow.

I've now added the two configuration parameters and it has solved the issue for me. Thank you for your input here.

The only issue I'm still facing is the parsing of the message where Header fields are mixed into the body. Consider the below sample message again:

8=FIXT.1.1|9=504|35=AE|487=0|1125=|1003=1120000338|75=20220210|1015=0|64=20220214|60=20220210-02:43:27.796|55=ANZ|48=AU000000ANZ3|22=4|461=Exxxxx|381=135000|31=27|32=5000.000000000000|15=AUD|1301=XASX|20003=XT|20007=XD|552=2|54=1|11=7533509260093758035|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|54=2|11=7533509260093757876|1=1040445|576=1|577=0|453=1|448=338-3|447=D|452=1|49=ASX|56=MSA01_3383_TEST|34=545|52=20220210-02:44:00.820|1128=9|880=7533509260093686098:0#NORMAL#1644451200000000000|167=CS|762=1|106=4075|10=087|

And the following code: https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/Message.java#L711 https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/Message.java#L773

The fields between Tags 552 and 452 is successfully processed by the parseGroup method. It is at the point where Tag49 is now considered, still within the parseGroup method. I believe the condition !(DataDictionary.HEADER_ID.equals(msgType)) should be changed (or at least enhanced) to !isHeaderField(tag) to allow Tag49 to be pushBack(field) as part of https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/Message.java#L778

and control may then be returned to the processBody method, where Tag49 will be correctly processed as part of the following condition:

https://github.com/quickfix-j/quickfixj/blob/48ea859f55904643a9059b91b37611e4424c6248/quickfixj-core/src/main/java/quickfix/Message.java#L681

I think a potential fix could be for Line 773:

if (!isTrailerField(tag) && !(DataDictionary.HEADER_ID.equals(msgType) || isHeaderField(tag))) {

So, DataDictionary.HEADER_ID.equals(msgType) would be true when processing a Repeating Group in the header , while isHeaderField(tag) would be true when processing a Header tag that was found in the body, directly following a repeating group.

Please let me know your thoughts on this.

Regards @jacques0803

[chrjohn]

Hi @jacques0803 thanks for your comment. I think your proposal makes sense. To make sure it does not break any existing functionality it would be appreciated If you could create a PR with your suggested change along with a unit test that shows the behaviour. Many thanks in advance!