sbe-python icon indicating copy to clipboard operation
sbe-python copied to clipboard

Handle implied padding in composite types and at end of blocks

Open threewholefish opened this issue 3 years ago • 2 comments

There was some code to handle implied padding, but it didn't quite go far enough for some functions.

The first type of padding is between composite type components, as indicated by the offset attribute of said components. The second is padding at the end of blocks to make the total length equal to the stated blockLength This happens for message fields (groups do not count towards the message blockLength) and group fields.

threewholefish avatar May 11 '22 17:05 threewholefish

Can you provide example data of when this is needed?

kizzx2 avatar May 15 '22 15:05 kizzx2

An example of padding within the composite is the following from the CME MDP3 spec:

<composite name="groupSize8Byte" description="8 Byte aligned repeating group dimensions" semanticType="NumInGroup">
    <type name="blockLength" description="Length" primitiveType="uint16"/>
    <type name="numInGroup" description="NumInGroup" offset="7" primitiveType="uint8"/>
</composite>

numInGroup has an offset of 7, meaning there are 5 implied padding bytes between it and blockLength

An example of padding at the end of blocks is the following from the same spec:

<group name="NoMDEntries" id="268" description="Number of Trade Summary entries" blockLength="32" dimensionType="groupSize">
    <field name="MDEntryPx" id="270" type="PRICE" description="Trade price" offset="0" semanticType="Price"/>
    <field name="MDEntrySize" id="271" type="Int32" description="Consolidated trade quantity" offset="8" semanticType="Qty"/>
    <field name="SecurityID" id="48" type="Int32" description="Security ID as defined by CME" offset="12" semanticType="int"/>
    <field name="RptSeq" id="83" type="uInt32" description="Sequence number per instrument update" offset="16" semanticType="int"/>
    <field name="NumberOfOrders" id="346" type="Int32NULL" description="The total number of real orders per instrument that participated in a match step within a match event" offset="20" semanticType="int"/>
    <field name="AggressorSide" id="5797" type="AggressorSide" description="Indicates which side is the aggressor or if there is no aggressor" offset="24" semanticType="int"/>
    <field name="MDUpdateAction" id="279" type="MDUpdateAction" description="Market Data update action" offset="25" semanticType="int"/>
    <field name="MDEntryType" id="269" type="MDEntryTypeTrade" description="Market Data entry type" semanticType="char"/>
    <field name="MDTradeEntryID" id="37711" type="uInt32NULL" description="Market Data Trade entry ID" offset="26" sinceVersion="7" semanticType="int"/>
</group>

This is the same problem as in #5, which is that the blockLength is greater than the sum of the lengths of the fields (32 and 30 respectively). The implied offset padding would also apply to fields within groups and within messages.

threewholefish avatar May 15 '22 17:05 threewholefish