questdb
questdb copied to clipboard
questdb
use tcp flow to write data to questdb database with 800 tables will out of memory
There are 800 tables in questdb databases,my system collect about 50,000 row per second. the questDB can only run about 30 minutes,used memory of the quest process will soon up to 180GB. the server.conf is default the OS: Centos7 the hardware is: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 196GB memory nvme 3T *2 the same data will ealy write to influxdb in same server.
We have memory telemetry in the latest (6.0.6) release. It would be useful to obtain that if that’s the version you are using
Before i used 6.0.5 can get this problem,Now the version update to 6.0.6,there also have this problem，how to open the the new version‘s memory telemetry function.
It is very primitive as of now, you can do select dump_memory_usage() and it will dump some memory breakdown to log file.
From top of my head I'd suggest decreasing cairo.inactive.writer.ttl in server.conf to something ridiculously low > 0, like 1.
How many columns are in total approximately in those 800 tables? Does QuestDB crash or log errors after the period of time?
It is very primitive as of now, you can do
select dump_memory_usage()and it will dump some memory breakdown to log file. From top of my head I'd suggest decreasingcairo.inactive.writer.ttlin server.conf to something ridiculously low > 0, like 1.How many columns are in total approximately in those 800 tables? Does QuestDB crash or log errors after the period of time?
every table of these 800 tables have 12~14 columns ,so total is about 10,000 columns. The memory used by QuestDB whill increase to server's max size ,and the os system will crash by out of memory .
when set cairo.inactive.writer.ttl=1，and runing for 1 hour minites the memory breakdown is:
2021-09-28T03:57:25.517204Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=613818856782, MMAP_DEFAULT=500766149230, NATIVE_DEFAULT=61178912, MMAP_O3=814237024, NATIVE_O3=56237228128, MMAP_TABLE_WRITER=53841068032, MMAP_TABLE_READER=5909632, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:03:59.413081Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=539055661298, MMAP_DEFAULT=434370885110, NATIVE_DEFAULT=62891612, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=44143419392, MMAP_TABLE_READER=667584, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:04:17.585273Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=541281975447, MMAP_DEFAULT=434936179662, NATIVE_DEFAULT=62901037, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=45804429312, MMAP_TABLE_READER=667836, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:04:41.556775Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=542998209418, MMAP_DEFAULT=435728999010, NATIVE_DEFAULT=63527748, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=46727217152, MMAP_TABLE_READER=667908, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:05:15.712846Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=546168380234, MMAP_DEFAULT=437959130796, NATIVE_DEFAULT=63993494, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=47666790400, MMAP_TABLE_READER=667944, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:05:38.639395Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=553918038391, MMAP_DEFAULT=444165105068, NATIVE_DEFAULT=64107935, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=49210359808, MMAP_TABLE_READER=667980, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:07:26.669517Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=580835555121, MMAP_DEFAULT=464801622474, NATIVE_DEFAULT=65737963, MMAP_O3=1030340400, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=53841068032, MMAP_TABLE_READER=543491180, NATIVE_FAST_MAP=8388608, NATIVE_LONG_LIST=67108864, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:07:33.165831Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=591147162737, MMAP_DEFAULT=474932120730, NATIVE_DEFAULT=65770731, MMAP_O3=461954016, NATIVE_O3=59134174752, MMAP_TABLE_WRITER=53841068032, MMAP_TABLE_READER=543491180, NATIVE_FAST_MAP=8388608, NATIVE_LONG_LIST=67108864, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:07:39.317937Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=570695864937, MMAP_DEFAULT=455692207154, NATIVE_DEFAULT=65803499, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=53841068032, MMAP_TABLE_READER=543491180, NATIVE_FAST_MAP=8388608, NATIVE_LONG_LIST=67108864, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:08:25.014204Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=556681524729, MMAP_DEFAULT=448886525576, NATIVE_DEFAULT=66963589, MMAP_O3=2373969776, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=44260859904, MMAP_TABLE_READER=539910812, NATIVE_FAST_MAP=8388608, NATIVE_LONG_LIST=67108864, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:10:11.923886Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=560262885408, MMAP_DEFAULT=449828940748, NATIVE_DEFAULT=68574592, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=49260691456, MMAP_TABLE_READER=551383540, NATIVE_FAST_MAP=8388608, NATIVE_LONG_LIST=67108864, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:11:09.594723Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=583313503004, MMAP_DEFAULT=471189447356, NATIVE_DEFAULT=68580496, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=50804260864, MMAP_TABLE_READER=622421744, NATIVE_FAST_MAP=16777216, NATIVE_LONG_LIST=134217728, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:11:15.425718Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=582507228860, MMAP_DEFAULT=467802655004, NATIVE_DEFAULT=68580496, MMAP_O3=2580518208, NATIVE_O3=58384711776, MMAP_TABLE_WRITER=50804260864, MMAP_TABLE_READER=622421744, NATIVE_FAST_MAP=16777216, NATIVE_LONG_LIST=134217728, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:11:23.644664Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=566956429004, MMAP_DEFAULT=453477442548, NATIVE_DEFAULT=68580496, MMAP_O3=518107048, NATIVE_O3=59221535536, MMAP_TABLE_WRITER=50804260864, MMAP_TABLE_READER=622421744, NATIVE_FAST_MAP=16777216, NATIVE_LONG_LIST=134217728, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:31:39.849557Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=622431751785, MMAP_DEFAULT=506963198148, NATIVE_DEFAULT=90622449, NATIVE_O3=60330868832, MMAP_TABLE_WRITER=52683390976, MMAP_TABLE_READER=119590612, NATIVE_FAST_MAP=16777216, NATIVE_LONG_LIST=134217728, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:35:44.826304Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=608164243531, MMAP_DEFAULT=486770425028, NATIVE_DEFAULT=95534843, MMAP_O3=5195908804, NATIVE_O3=61874372704, MMAP_TABLE_WRITER=50418368512, MMAP_TABLE_READER=1452306664, NATIVE_FAST_MAP=29360128, NATIVE_LONG_LIST=234881024, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:37:54.247800Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=632862167305, MMAP_DEFAULT=508901752794, NATIVE_DEFAULT=98227691, MMAP_O3=4486195552, NATIVE_O3=61874372704, MMAP_TABLE_WRITER=53690064896, MMAP_TABLE_READER=1454226692, NATIVE_FAST_MAP=29360128, NATIVE_LONG_LIST=234881024, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:56:54.510438Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=611077590409, MMAP_DEFAULT=487294147410, NATIVE_DEFAULT=130691687, MMAP_O3=8162101864, NATIVE_O3=60532195424, MMAP_TABLE_WRITER=51961937920, MMAP_TABLE_READER=525942920, NATIVE_FAST_MAP=41943040, NATIVE_LONG_LIST=335544320, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T04:58:54.169652Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=573365584514, MMAP_DEFAULT=455113669544, NATIVE_DEFAULT=132664022, MMAP_O3=6559504252, NATIVE_O3=58988691552, MMAP_TABLE_WRITER=46911774720, MMAP_TABLE_READER=3037712296, NATIVE_FAST_MAP=58720256, NATIVE_LONG_LIST=469762048, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T05:22:22.652336Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=604091101501, MMAP_DEFAULT=460081959292, NATIVE_DEFAULT=145365337, MMAP_O3=3225021232, NATIVE_O3=81352720480, MMAP_TABLE_WRITER=51508928512, MMAP_TABLE_READER=5004543576, NATIVE_FAST_MAP=75497472, NATIVE_LONG_LIST=603979776, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T05:22:48.882525Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=602017541061, MMAP_DEFAULT=460081959292, NATIVE_DEFAULT=145365337, MMAP_O3=1151460792, NATIVE_O3=81352720480, MMAP_TABLE_WRITER=51508928512, MMAP_TABLE_READER=5004543576, NATIVE_FAST_MAP=75497472, NATIVE_LONG_LIST=603979776, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T05:23:52.238073Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=572129991719, MMAP_DEFAULT=434400650864, NATIVE_DEFAULT=145373471, MMAP_O3=1347643120, NATIVE_O3=82522162160, MMAP_TABLE_WRITER=47129886720, MMAP_TABLE_READER=3811712312, NATIVE_FAST_MAP=75497472, NATIVE_LONG_LIST=603979776, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
2021-09-28T05:24:07.921045Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=572129991719, MMAP_DEFAULT=434400650864, NATIVE_DEFAULT=145373471, MMAP_O3=1347643120, NATIVE_O3=82522162160, MMAP_TABLE_WRITER=47129886720, MMAP_TABLE_READER=3811712312, NATIVE_FAST_MAP=75497472, NATIVE_LONG_LIST=603979776, NATIVE_HTTP_CONN=616690816, NATIVE_PGW_CONN=1476395008]
the used memory is 180GB
[root@sh-tsdb8 conf]# ps -aux |grep 37045
root 37045 723 93.3 760367684 183593620 pts/3 Sl 11:43 733:06 ./java -DQuestDB-Runtime-66535 -ea -Dnoebug -XX:+UnlockExperimentalVMOptions -XX:+AlwaysPreTouch -XX:+UseParallelOldGC -Xms16G -Xmx16G -XX:MaxHeapFreeRatio=40 -p ./questdb.jar -m io.questdb/io.questdb.ServerMain -d /srv/1/questdb
[root@sh-tsdb8 conf]#
this is top command result：
The default configuration allocates around 32+Mb of RAM per column. In your case of 10k columns it results to 320Gb total RAM reserved for writing only.
It is excessive and we can do few thing to cut the memory usage, especially in the case when there are many small tables to write. Unfortunately atm the buffer sizes for writing are hard coded
I'm opening a feature request to make this configurable
@lg31415 Thanks for submitting the issue, it was very useful to realise that the memory usage is far from optimal. Sorry I couldn't provide you an immediate solution or workaround
for test this 320GB memory way can improve my problem，I change my server with 396GB memory.
after running for 1 hour, the questDB seem that lost some data from by tcp input when the memory has used 265GB.
at last the memroy used is 337GB,this time the questDB also input some data.
it seem strange, there no quest by client at all ,but when write data to disk there also some read data from disk .
the dump_memory_usage result is
2021-09-30T04:01:11.898451Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=486548146521, MMAP_DEFAULT=350947699586, NATIVE_DEFAULT=116957794, MMAP_O3=7614161656, NATIVE_O3=69021466752, MMAP_TABLE_WRITER=52951842816, MMAP_TABLE_READER=2192276605, NATIVE_FAST_MAP=96468992, NATIVE_LONG_LIST=771751936, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:01:12.564887Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=478933984865, MMAP_DEFAULT=350947699586, NATIVE_DEFAULT=116957794, NATIVE_O3=69021466752, MMAP_TABLE_WRITER=52951842816, MMAP_TABLE_READER=2192276605, NATIVE_FAST_MAP=96468992, NATIVE_LONG_LIST=771751936, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:01:35.167850Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=489673316374, MMAP_DEFAULT=359274937256, NATIVE_DEFAULT=116966633, MMAP_O3=1304654440, NATIVE_O3=69021466752, MMAP_TABLE_WRITER=54059188224, MMAP_TABLE_READER=2192361757, NATIVE_FAST_MAP=96468992, NATIVE_LONG_LIST=771751936, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:03:25.084428Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=507124682309, MMAP_DEFAULT=368269127936, NATIVE_DEFAULT=117009908, MMAP_O3=9264448264, NATIVE_O3=69021466752, MMAP_TABLE_WRITER=54243745792, MMAP_TABLE_READER=2429644873, NATIVE_FAST_MAP=104857600, NATIVE_LONG_LIST=838860800, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:06:35.154807Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=483339561170, MMAP_DEFAULT=354850136852, NATIVE_DEFAULT=117684450, NATIVE_O3=67947724928, MMAP_TABLE_WRITER=49562689536, MMAP_TABLE_READER=6704599260, NATIVE_FAST_MAP=146800640, NATIVE_LONG_LIST=1174405120, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:07:13.623791Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=489448878320, MMAP_DEFAULT=350501317824, NATIVE_DEFAULT=118472624, MMAP_O3=9449955652, NATIVE_O3=68954357888, MMAP_TABLE_WRITER=49562689536, MMAP_TABLE_READER=6705358652, NATIVE_FAST_MAP=146800640, NATIVE_LONG_LIST=1174405120, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:41:11.921886Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=476463620352, MMAP_DEFAULT=344022043478, NATIVE_DEFAULT=121586286, MMAP_O3=74604720, NATIVE_O3=75564580992, MMAP_TABLE_WRITER=49831141376, MMAP_TABLE_READER=2353198732, NATIVE_FAST_MAP=184549376, NATIVE_LONG_LIST=1476395008, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:41:29.658174Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=478391165089, MMAP_DEFAULT=346024006238, NATIVE_DEFAULT=121587459, NATIVE_O3=75564580992, MMAP_TABLE_WRITER=49831141376, MMAP_TABLE_READER=2353384256, NATIVE_FAST_MAP=184549376, NATIVE_LONG_LIST=1476395008, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T04:41:46.212837Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=479768226945, MMAP_DEFAULT=345856943582, NATIVE_DEFAULT=121587459, MMAP_O3=1544080032, NATIVE_O3=75564580992, MMAP_TABLE_WRITER=49831141376, MMAP_TABLE_READER=2353428736, NATIVE_FAST_MAP=184549376, NATIVE_LONG_LIST=1476395008, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T05:17:02.809930Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=493515139159, MMAP_DEFAULT=353157665216, NATIVE_DEFAULT=121695007, NATIVE_O3=82879447168, MMAP_TABLE_WRITER=49059348480, MMAP_TABLE_READER=3725021048, NATIVE_FAST_MAP=192937984, NATIVE_LONG_LIST=1543503872, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T05:17:04.230731Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=493246532531, MMAP_DEFAULT=352889058588, NATIVE_DEFAULT=121695007, NATIVE_O3=82879447168, MMAP_TABLE_WRITER=49059348480, MMAP_TABLE_READER=3725021048, NATIVE_FAST_MAP=192937984, NATIVE_LONG_LIST=1543503872, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
2021-09-30T05:24:57.521490Z I i.q.g.e.f.c.DumpMemoryUsageFunctionFactory$DumpMemoryUsage memory usage [TOTAL=485200179311, MMAP_DEFAULT=346011046738, NATIVE_DEFAULT=121611769, MMAP_O3=298810016, NATIVE_O3=82476793984, MMAP_TABLE_WRITER=51626369024, MMAP_TABLE_READER=93585540, NATIVE_FAST_MAP=192937984, NATIVE_LONG_LIST=1543503872, NATIVE_HTTP_CONN=822254464, NATIVE_PGW_CONN=2013265920]
Thanks a lot for this information. Could we drill down into data loss please?
Could you post errors from questdb log? Grep for ‘ E ‘
Thanks a lot for this information. Could we drill down into data loss please?
Could you post errors from questdb log? Grep for ‘ E ‘
All Error log is:
2021-09-30T02:09:09.847477Z E i.q.TelemetryJob instance [id=0x05cd2cebd45efa000003150609f2b4, enabled=true]
2021-09-30T02:09:11.818709Z E i.q.c.l.t.LineTcpConnectionContext [8047] could not process line data
2021-09-30T02:09:11.979278Z E i.q.c.l.t.LineTcpConnectionContext [29712] could not process line data
2021-09-30T02:09:13.580145Z E i.q.c.l.t.LineTcpConnectionContext [52234] could not process line data
2021-09-30T02:09:13.777086Z E i.q.c.l.t.LineTcpConnectionContext [52249] could not process line data
2021-09-30T02:09:13.906945Z E i.q.c.l.t.LineTcpConnectionContext [52257] could not process line data
2021-09-30T02:09:17.617787Z E i.q.c.l.t.LineTcpConnectionContext [64007] could not process line data
2021-09-30T02:09:18.697859Z E i.q.c.l.t.LineTcpConnectionContext [64156] could not process line data
2021-09-30T02:09:22.872760Z E i.q.c.l.t.LineTcpConnectionContext [65892] could not process line data
2021-09-30T02:09:22.882196Z E i.q.c.l.t.LineTcpConnectionContext [65786] could not process line data
2021-09-30T02:09:22.922344Z E i.q.c.l.t.LineTcpConnectionContext [65890] could not process line data
2021-09-30T02:09:22.985102Z E i.q.c.l.t.LineTcpConnectionContext [65893] could not process line data
2021-09-30T02:09:23.222783Z E i.q.c.l.t.LineTcpConnectionContext [66625] could not process line data
2021-09-30T02:09:23.360938Z E i.q.c.l.t.LineTcpConnectionContext [67008] could not process line data
2021-09-30T02:09:23.460805Z E i.q.c.l.t.LineTcpConnectionContext [65893] could not process line data
2021-09-30T02:09:23.503431Z E i.q.c.l.t.LineTcpConnectionContext [68021] could not process line data
2021-09-30T02:09:23.609420Z E i.q.c.l.t.LineTcpConnectionContext [67310] could not process line data
2021-09-30T02:09:32.953613Z E i.q.c.l.t.LineTcpConnectionContext [72997] could not process line data
2021-09-30T02:10:04.390222Z E i.q.c.l.t.LineTcpConnectionContext [79459] could not process line data
2021-09-30T02:10:04.621068Z E i.q.c.l.t.LineTcpConnectionContext [79314] could not process line data
2021-09-30T02:10:05.239658Z E i.q.c.l.t.LineTcpConnectionContext [79559] could not process line data
2021-09-30T02:10:05.255187Z E i.q.c.l.t.LineTcpConnectionContext [79800] could not process line data
2021-09-30T02:10:57.007830Z E i.q.c.l.t.LineTcpConnectionContext [71718] could not process line data
2021-09-30T02:11:49.244596Z E i.q.c.l.t.LineTcpConnectionContext [622] could not process line data
2021-09-30T02:12:50.866865Z E i.q.c.l.t.LineTcpConnectionContext [39955] could not process line data
2021-09-30T02:12:55.038550Z E i.q.c.l.t.LineTcpConnectionContext [64336] could not process line data
2021-09-30T02:16:54.634291Z E i.q.c.l.t.LineTcpConnectionContext [57845] could not process line data
2021-09-30T02:19:04.914928Z E i.q.c.l.t.LineTcpConnectionContext [70941] could not process line data
2021-09-30T02:28:08.361601Z E i.q.c.l.t.LineTcpConnectionContext [77291] could not process line data
2021-09-30T02:33:54.335948Z E i.q.c.l.t.LineTcpConnectionContext [2114] could not process line data
2021-09-30T02:43:24.878493Z E i.q.c.l.t.LineTcpConnectionContext [81044] could not process line data
2021-09-30T02:58:36.042916Z E i.q.c.l.t.LineTcpConnectionContext [58839] could not process line data
2021-09-30T03:06:30.116145Z E i.q.c.l.t.LineTcpConnectionContext [77177] could not process line data
2021-09-30T03:20:56.663414Z E i.q.c.l.t.LineTcpConnectionContext [22291] could not process line data
2021-09-30T03:23:15.983967Z E i.q.c.l.t.LineTcpConnectionContext [45159] could not process line data
2021-09-30T03:36:18.411071Z E i.q.c.O3PurgeDiscoveryJob queuing [table=cdn.bus.sink.flow.width, ts=2021-09-30T00:00:00.000000Z, txn=863280, errno=-1]
2021-09-30T03:43:46.700868Z E i.q.c.l.t.LineTcpConnectionContext [7495] could not process line data
2021-09-30T03:59:48.303583Z E i.q.c.l.t.LineTcpConnectionContext [36420] could not parse measurement, code NO_FIELDS at 5 line (may be mangled due to partial parsing) is
2021-09-30T04:08:48.462165Z E i.q.c.p.PGConnectionContext error [pos=7, msg=`not enough columns in group by`, errno=`0]
2021-09-30T04:09:00.152725Z E i.q.c.p.PGConnectionContext error [pos=7, msg=`not enough columns in group by`, errno=`0]
How do you run questdb? Docker or binary?
binary，the version is 6.0.7，package is questdb-6.0.7-rt-linux-amd64
Hi @lg31415 it might be more efficient to discuss further via other messaging channels, are you already in QuestDB's community slack? you can join via https://slack.questdb.io
Could we have full log from fail run ? We also need context around the error log lines to understand what the issue could be
there is PR in ready to go to tackle high memory consumption:
https://github.com/questdb/questdb/pull/1373
Could we have full log from fail run ? We also need context around the error log lines to understand what the issue could be
Could we have full log from fail run ? We also need context around the error log lines to understand what the issue could be
2021-09-30T02:09:11.906622Z I tcp-line-server disconnected [ip=39.135.1.2, fd=16385, src=queue]^M
2021-09-30T02:09:11.907943Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=3, tableName=cdn.bus.sink.origin.request.code.444.host, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.912080Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.request.code.500.host, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.913514Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.origin.request.code.500.host, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.916592Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.origin.request.code.500, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.918482Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.origin.request.code.307.host, nNetworkIoWorkers=6]^M
2021-09-30T02:09:11.920441Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.origin.request.code.424, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.921340Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.request.code.424.host, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.922290Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=2, tableName=cdn.bus.sink.origin.request.code.424.host, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.929091Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.origin.request.code.304.host, nNetworkIoWorkers=4]^M
2021-09-30T02:09:11.935109Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=0, tableName=cdn.bus.sink.origin.request.code.304, nNetworkIoWorkers=5]^M
2021-09-30T02:09:11.936219Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.request.code.400, nNetworkIoWorkers=5]^M
2021-09-30T02:09:11.936932Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.origin.request.code.400, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.942584Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=0, tableName=cdn.bus.sink.origin.request.code.304.host, nNetworkIoWorkers=5]^M
2021-09-30T02:09:11.943907Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.origin.request.code.400.host, nNetworkIoWorkers=4]^M
2021-09-30T02:09:11.944560Z I tcp-line-server scheduling disconnect [fd=26031, reason=0]^M
2021-09-30T02:09:11.944627Z I tcp-line-server disconnected [ip=39.135.1.2, fd=26031, src=queue]^M
2021-09-30T02:09:11.944728Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=0, tableName=cdn.bus.sink.origin.request.code.500.host, nNetworkIoWorkers=4]^M
2021-09-30T02:09:11.945105Z I tcp-line-server connected [ip=39.135.1.2, fd=44753]^M
2021-09-30T02:09:11.946281Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=0, tableName=cdn.bus.sink.request.code.500, nNetworkIoWorkers=2]^M
2021-09-30T02:09:11.954096Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=1, tableName=cdn.bus.sink.request.code.500, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.956162Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=4, tableName=cdn.bus.sink.origin.request.code.500, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.956863Z I tcp-line-server scheduling disconnect [fd=27843, reason=0]^M
2021-09-30T02:09:11.956884Z I tcp-line-server disconnected [ip=39.135.1.2, fd=27843, src=queue]^M
2021-09-30T02:09:11.961895Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=3, tableName=cdn.bus.sink.request.code.410.host, nNetworkIoWorkers=6]^M
2021-09-30T02:09:11.966424Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=5, tableName=cdn.bus.sink.origin.request.code.400, nNetworkIoWorkers=4]^M
2021-09-30T02:09:11.968567Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=5, tableName=cdn.bus.sink.request.code.500.host, nNetworkIoWorkers=3]^M
2021-09-30T02:09:11.970939Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=5, tableName=cdn.bus.sink.origin.request.code.304, nNetworkIoWorkers=6]^M
2021-09-30T02:09:11.974869Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=5, tableName=cdn.bus.sink.request.code.307, nNetworkIoWorkers=6]^M
2021-09-30T02:09:11.975924Z I tcp-line-server scheduling disconnect [fd=28043, reason=0]^M
2021-09-30T02:09:11.976105Z I tcp-line-server disconnected [ip=39.135.1.2, fd=28043, src=queue]^M
2021-09-30T02:09:11.979278Z E i.q.c.l.t.LineTcpConnectionContext [29712] could not process line data^M
io.questdb.cairo.CairoException: [0] String is outside of file boundary [offset=89756, len=19, size=89756]^M
^M
2021-09-30T02:09:11.979350Z I tcp-line-server scheduling disconnect [fd=29712, reason=0]^M
2021-09-30T02:09:11.981621Z I i.q.c.l.t.LineTcpMeasurementScheduler network IO thread using table [workerId=1, tableName=cdn.bus.sink.origin.request.code.500, nNetworkIoWorkers=4]^M
2021-09-30T02:09:11.981808Z I tcp-line-server disconnected [ip=39.135.1.2, fd=29712, src=queue]^M
it seem that the problem is String is outside of file boundary ?
thanks, I've seen this String is outside of file boundary elsewhere too. Is this new set of tables you're inserting to or have they been upgraded from previous QuestDB version ?
thanks, I've seen this
String is outside of file boundaryelsewhere too. Is this new set of tables you're inserting to or have they been upgraded from previous QuestDB version ?
The table is created at the beginning, and I have deleted the data directory, so these tables are new.
Hi @lg31415 Are you still experiencing this issue with the latest version? (https://github.com/questdb/questdb/releases)
We added configuration parameters for cases when there is large number of small tables, please see #1369
I am still getting this error and have no idea why. I'm a new user to questdb and posted in the Slack channel as well
I'm able to ingest 6B rows in total into 4K WAL tables on the latest master (which will be shipped soon) with the following settings:
cairo.writer.data.append.page.size=32K
cairo.o3.column.memory.size=32K
cairo.writer.data.index.key.append.page.size=32K
cairo.writer.data.index.value.append.page.size=32K
line.default.partition.by=MONTH
cairo.max.uncommitted.rows=10000
cairo.wal.commit.squash.row.limit=10000
cairo.wal.segment.rollover.row.count=10000
line.tcp.min.idle.ms.before.writer.release=60000000
cairo.inactive.writer.ttl=60000000
cairo.wal.inactive.writer.ttl=60000000
The data is generated with TSBS' cpu use case, but with a modification to use a distinct table per each hostname.
Closing this one as it seems no longer relevant.
