rdf -> HDT process killed. Out of memory

[ahmad88me]

Hi,

I have a server with 15 GB of RAM and a ttl of (~13 GB). It tried to generate HDT from the ttl but after some time the process just get killed by the OS (debian) with the following message (using dmesg | less):

Out of memory: Kill process 22472 (rdf2hdt) score 979 or sacrifice child

Is there a way around this without increasing the RAM? Is there a way to for example split the ttl into smaller chunks, convert each chunk to a separate HDT and then merge the HDT files together? or maybe append a chunk of ttl to an existing HDT?

Thanks in Advance.

[ahmad88me]

I found the mergeHDT. So I did another trial. I split the files into smaller sizes (<4 GB each). I converted them to HDT with no problem. Then when I tried to merge the HDTs, I encountered memory issue (bad malloc, which I think is related to a similar memory issue).

[mielvds]

Indeed, 13gb will overflow the memory. Maybe @wouterbeek has some suggestions from LODalot? Not sure what merge implementation you are referring to, but the Java version has hdtCat and that should be able to do these merges.

[ahmad88me]

Thanks for your reply. I am referring to this merge (https://github.com/rdfhdt/hdt-cpp/blob/332a9cc2d5273e76b9daad366f7d2f80adb6b3fc/libhdt/tests/mergeHDT.cpp)