quickfixj icon indicating copy to clipboard operation
quickfixj copied to clipboard

Published 20 hours ago • verified publisher icon quickfix-j

if config ResendRequestChunkSize arg. is didn't work，probably oom

Open wajncn opened this issue 1 year ago • 15 comments
trafficstars

Describe the bug If the amount of data is particularly large 。

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N

in quickfixj-initiator delete local quickfixj file。 restart quickfixj-acceptor OOM （If there is no excessive memory）

need config

ResendRequestChunkSize=1000

Can solve this problem

system information:

  • OS: Win10
  • Java version JDK8
  • QFJ Version FIX.5.0SP2

https://github.com/quickfix-j/quickfixj/pull/777

wajncn avatar Mar 18 '24 11:03 wajncn

image

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N
DisconnectOnError=N
ContinueInitializationOnError=N
ResendRequestChunkSize=3

INFO quickfix.Session - load beginSeqNo1,endSeqNo:3  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo4,endSeqNo:6  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo7,endSeqNo:9  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo10,endSeqNo:12  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo12,endSeqNo:14  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo15,endSeqNo:17  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo18,endSeqNo:20  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo21,endSeqNo:23  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo24,endSeqNo:26  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo27,endSeqNo:29  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo30,endSeqNo:32  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo33,endSeqNo:35  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo36,endSeqNo:38  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo39,endSeqNo:41  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo42,endSeqNo:44  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo45,endSeqNo:47  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo48,endSeqNo:50  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo51,endSeqNo:53  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo54,endSeqNo:56  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo57,endSeqNo:59  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo60,endSeqNo:62  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo63,endSeqNo:65  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo66,endSeqNo:68  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo69,endSeqNo:71  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo72,endSeqNo:74  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo75,endSeqNo:77  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo78,endSeqNo:80  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo81,endSeqNo:83  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo84,endSeqNo:86  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo87,endSeqNo:89  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo90,endSeqNo:92  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo93,endSeqNo:95  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo95,endSeqNo:97  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo98,endSeqNo:100  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo101,endSeqNo:103  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo104,endSeqNo:106  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo107,endSeqNo:109  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo110,endSeqNo:112  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo113,endSeqNo:115  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo116,endSeqNo:118  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo119,endSeqNo:121  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo122,endSeqNo:124  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo125,endSeqNo:127  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo128,endSeqNo:130  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo131,endSeqNo:133  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo134,endSeqNo:136  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo137,endSeqNo:139  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo140,endSeqNo:142  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo142,endSeqNo:144  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo145,endSeqNo:147  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo148,endSeqNo:150  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo151,endSeqNo:153  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo154,endSeqNo:156  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo157,endSeqNo:159  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo160,endSeqNo:162  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo163,endSeqNo:165  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo166,endSeqNo:168  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo169,endSeqNo:171  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo172,endSeqNo:174  beginString：FIXT.1.1

wajncn avatar Mar 18 '24 11:03 wajncn

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

chrjohn avatar Mar 18 '24 15:03 chrjohn

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

If you don't change the code， endSeqNo is Max SeqNo。 FIleStore all the data is loaded。
Although the parameters are configured 【ResendRequestChunkSize=3】 。

So, change the code。

wajncn avatar Mar 18 '24 16:03 wajncn

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

Suppose the store file is 3GB，100000rows。 java max heap is 2G。
acceptor and initiator config【ResendRequestChunkSize=1000】。 acceptor online。 initiator offline。 then start initiator after。 acceptor need load 3GBdata to memory，bug memory limit 2G。 then OOM exception～ 。

if merge this issues，there wouldn't be this problem。

wajncn avatar Mar 18 '24 16:03 wajncn

OK, starting to understand. Are you able to supply a unit test?

chrjohn avatar Mar 18 '24 17:03 chrjohn

OK, starting to understand. Are you able to supply a unit test?

I wrote it locally acceptor and initiator demo。 It meets my expectations

wajncn avatar Mar 19 '24 03:03 wajncn

OK, starting to understand. Are you able to supply a unit test? I submit a unit test as you requested 。 you can debug 【Session#2293 line https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2293】 and 【quickfix.Session#resendMessages】 【https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2335】

wajncn avatar Mar 19 '24 05:03 wajncn

OK, starting to understand. Are you able to supply a unit test?

hi ，Please help me look at this pr。

wajncn avatar Mar 20 '24 13:03 wajncn

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

chrjohn avatar Mar 20 '24 18:03 chrjohn

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

how test passed ？

wajncn avatar Mar 22 '24 14:03 wajncn

??

wajncn avatar Apr 02 '24 13:04 wajncn

https://github.com/quickfix-j/quickfixj/issues/778#issuecomment-2010365727

:)

chrjohn avatar Apr 02 '24 14:04 chrjohn

oh....

wajncn avatar May 23 '24 02:05 wajncn

merge ？

wajncn avatar May 23 '24 02:05 wajncn

Did you see my comment?
https://github.com/quickfix-j/quickfixj/pull/777#discussion_r1548045908

chrjohn avatar May 23 '24 03:05 chrjohn