quickfix icon indicating copy to clipboard operation
quickfix copied to clipboard

Support SaveMessagesAndIncrNextSenderMsgSeqNum in MessageStore

Open Yu-Xie opened this issue 2 years ago • 0 comments

Today, MessageStore persist message to send one by one through this method:

SaveMessageAndIncrNextSenderMsgSeqNum(seqNum int, msg []byte) error

This means each outgoing message maps to one database operation. This is less efficient when there is traffic spikes. I did a benchmark using SQL Store, with local laptop and Postgres in Docker. If you send out X outgoing messages all at once by calling SendToTarget, the results are listed below (which is very slow because we write to database for every single message without batching at all):

  • 100 Messages = 204 ms (490 QPS)
  • 1000 Messages = 1568 ms (638 QPS)
  • 5000 Messages = 8294 ms (602 QPS)
  • 10000 Messages = 15712 ms (636 QPS)

The proposal is to support batching for outgoing messages:

  1. Support a bulk method func SaveMessagesAndIncrNextSenderMsgSeqNum(seqNum int, msg [][]byte) in the storage interface.
  2. Support a bulk method func SendAppToTarget(m []Messagable, sessionID SessionID) error in registry.go which invokes 1. underneath.

Prototype is done in this PR https://github.com/quickfixgo/quickfix/pull/576 with SQLStore and In Memory Store support (not file store / mongo store yet).

@ackleymi @mgatny What do you think?

Yu-Xie avatar Apr 27 '23 03:04 Yu-Xie