TarsosLSH icon indicating copy to clipboard operation
TarsosLSH copied to clipboard

Published 20 hours ago • verified publisher icon JorenSix

GC overhead limit exceeded

Open jatinagarwal opened this issue 8 years ago • 2 comments

Even after using 13g of memory I get "GC overhead limit exceeded" exception with following stack trace: Picked up _JAVA_OPTIONS: -Xms2g -Xmx13g -Xss100M -XX:MaxPermSize=3g Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.(String.java:203) at java.lang.String.substring(String.java:1913) at java.lang.String.split(String.java:2288) at java.lang.String.split(String.java:2355) at be.tarsos.lsh.util.FileUtils.readCSVFile(Unknown Source) at be.tarsos.lsh.LSH.readDataset(Unknown Source) at be.tarsos.lsh.CommandLineInterface.parseArguments(Unknown Source) at be.tarsos.lsh.LSH.main(Unknown Source)

My dataset size is 4GB, it contains 15201 vectors. I run with following options "java -jar TarsosLSH-0.7.jar -f l2 -h 8 -t 32 -n 15 -d tfVectorFor105G -q tfVectorQueriesOn105g"

jatinagarwal avatar Oct 21 '15 06:10 jatinagarwal

This seems indeed to be a problem related to memory. The -X params should be able to correct things. How large (dimensions) are your vectors?

JorenSix avatar Oct 22 '15 11:10 JorenSix

Vectosr have 66669 dimensions.

jatinagarwal avatar Oct 22 '15 11:10 jatinagarwal