error reading collection

[StefanoSpada]

Hi, I'm trying to read in R, in chunks of 50k rows at a time, a collection of about 300k rows.

  dt <- dbGetQuery(mongo, arg, querystr,(i*chunk_size),chunk_size)

This is the error I get back Error in .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, : java.lang.OutOfMemoryError: Java heap space

The system has still plenty of memory available and the user I'm running the has the ulimit set to unlimited memory allocation.

[tc]

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see: http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

[StefanoSpada]

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [email protected] wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-44988480.

Stefano Spada CEO & Founder GRANDSLA, Inc. http://www.grandsla.com skype: stefano.spada g+:[email protected] +1 650 391 4450

[tc]

Did increasing the heap space by setting the ENV variable resolve the issue?

On Tue, Jun 3, 2014 at 10:54 AM, StefanoSpada [email protected] wrote:

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [email protected] wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-44988480.

Stefano Spada CEO & Founder GRANDSLA, Inc. http://www.grandsla.com skype: stefano.spada g+:[email protected] +1 650 391 4450

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-44998161.

Tommy Chheng

[StefanoSpada]

I increased the java heap as suggested by setting JAVA_OPTS="-Xms512m -Xmx1g"

still I get the error ...

Error in .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, : java.lang.OutOfMemoryError: Java heap space 16 stop(structure(list(message = "java.lang.OutOfMemoryError: Java heap space", call = .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, query, keys, skip, limit), jobj = <S4 object of class structure("jobjRef", package = "rJava")>), .Names = c("message", "call", "jobj"), class = c("OutOfMemoryError", "VirtualMachineError", "Error", "Throwable", "Object", "Exception", "error", "condition" ))) 15 .jcheck(silent = FALSE) 14 .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, query, keys, skip, limit) 13 dbGetQueryForKeys(rmongo.object, collection, query, "{}", skip, limit) 12 dbGetQueryForKeys(rmongo.object, collection, query, "{}", skip, limit) 11 dbGetQuery(mongo, arg, querystr, (i * chunk_size), chunk_size) 10 dbGetQuery(mongo, arg, querystr, (i * chunk_size), chunk_size) at rserveSource.R#55 9 connect_rmongo(collection) at rserveSource.R#97 8 eval(expr, envir, enclos) 7 eval(ei, envir) 6 withVisible(eval(ei, envir)) 5 source("/home/stefano/git/server/server-mgmt/rserveSource.R") at 2mongo.R#3 4 eval(expr, envir, enclos) 3 eval(ei, envir) 2 withVisible(eval(ei, envir)) 1 source("2mongo.R")

On Tue, Jun 3, 2014 at 3:25 PM, Tommy Chheng [email protected] wrote:

Did increasing the heap space by setting the ENV variable resolve the issue?

On Tue, Jun 3, 2014 at 10:54 AM, StefanoSpada [email protected] wrote:

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [email protected] wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-44988480.

Stefano Spada CEO & Founder GRANDSLA, Inc. http://www.grandsla.com skype: stefano.spada g+:[email protected] +1 650 391 4450

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-44998161.

Tommy Chheng

— Reply to this email directly or view it on GitHub https://github.com/tc/RMongo/issues/30#issuecomment-45028507.

Stefano Spada CEO & Founder GRANDSLA, Inc. http://www.grandsla.com skype: stefano.spada g+:[email protected] +1 650 391 4450

[tc]

According to http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

You need to set JAVA_TOOL_OPTIONS or _JAVA_TOOL_OPTIONS, not JAVA_OPTS