FR: Allow specifying JVM flags for Java-based emulators

[olivierkrener]

[REQUIRED] Environment info

firebase-tools: 8.6.0

Platform: Windows 10, Chrome 84.0.4147.105

[REQUIRED] Test case

I have no instructions to reproduce this issue, but I can describe the circumstances of it occurring

I have a large ruleset (around 60Ko) and many tests (200+) covering most of my ruleset.

When trying to visualize the report at http://localhost:8080/emulator/v1/projects/my-firebase-project:ruleCoverage I will get an error "sometimes", the only solution is then to restart the emulator and hope it will work the next time.

It seems that the report generation in Chrome is exhausting the capabilities of my laptop...

[REQUIRED] Steps to reproduce

See above

[REQUIRED] Expected behavior

Report should be generated

[REQUIRED] Actual behavior

http://localhost:8080/emulator/v1/projects/my-firebase-project:ruleCoverage {"error":{"code":500,"message":"Internal","status":"INTERNAL"}}

firestore-debug.log Aug 04, 2020 9:52:10 AM io.gapi.emulators.netty.HttpHandler$1 onError INFO: Exception when handling request: Java heap space

How would I increase the "Java heap space" for the emulator?

[samtstern]

@olivierkrener thanks for reporting! Sounds like a bug in the Firestore emulator. Assigning to @samhorlbeck who is on call at the moment.

[sam-gc]

Thanks! I've filed a bug (for internal folks: b/162846002). In the meantime @yuchenshi may know how to increase the JVM heap when starting the emulator

[scottcrossen]

I just want to comment to say "nice". Well done for finding this limit!

[yuchenshi]

The java process heap size can be specified using the JVM option -Xmx256m (or Xmx1024m for bigger, ditto).

We however don't have a way to pass that argument from the Firebase CLI. For now, you can manually start the emulator by invoking the jar file manually. It's along the lines of java -jar ~/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.5.jar on Linux / OS X. (You can spin up the Emulator Suite using the CLI first, and then use ps aux | grep java to see the full command-line.) I have less idea about Windows, but I assume you can just look through all processes with the full command-line using PowerShell or CMD while the Emulator Suite is running.

Once you can start the jar manually, try adding JVM options above to the command-line and see if that fixes your problem. If that works, please file a feature request to improve the ergonomics, for example, by allowing specifying heap sizes through Firebase CLI flags or firebase.json.

[olivierkrener]

I cant seem to have the emulator properly running via the command line. This is the one I found by browsing the process in PowerShell:

java -Duser.language=en -jar pathtoemulator\cloud-firestore-emulator-v1.11.5.jar --host localhost --port 8080 --rules pathtorules\firestore.rules node pathtoemulator\ui-v1.1.1\server.bundle.js (pathtoemulator and pathtorules to be replaced of course)

The output I get when running this command is

API endpoint: http://localhost:8080 If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:

export FIRESTORE_EMULATOR_HOST=localhost:8080

Dev App Server is now running.

Aug 05, 2020 11:42:48 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. Aug 05, 2020 11:42:48 AM io.gapi.emulators.netty.NotFoundHandler handleRequest INFO: Unknown request URI: /service-worker.js Aug 05, 2020 11:43:18 AM io.gapi.emulators.netty.NotFoundHandler handleRequest INFO: Unknown request URI: /service-worker.js Aug 05, 2020 11:43:48 AM io.gapi.emulators.netty.NotFoundHandler handleRequest INFO: Unknown request URI: /service-worker.js ...

And my test suite can't connect to it. Any idea how I could proceed?

[yuchenshi]

I don't see anything wrong with the startup and it should be listening. Please make sure the process is running in another terminal while you run the tests. Also: Did you set the environment variables and all that in your test setup? Does your tests need other emulators running as well? Please show a snippet of how you connect to the emulator with testing.

Also, you may need to change host to 0.0.0.0 or 127.0.0.1 instead of localhost or try another port. (If you're using WSL, be aware there are more issues involved https://github.com/firebase/firebase-tools-ui/issues/332 so I'd suggest you run java on native Windows CMD if possible.)

[olivierkrener]

Please` make sure the process is running in another terminal while you run the tests

The process is running in another terminal

Also:` Did you set the environment variables and all that in your test setup?

No, nothing at all. What variables do I need to setup?

Does` your tests need other emulators running as well? Please show a snippet of how you connect to the emulator with testing.

Usually I just start the emulator via the firebase library using the command: firebase emulators:start --only firestore

Also, you may need to change host to 0.0.0.0 or 127.0.0.1 instead of localhost or try another port. (If you're using WSL, be aware there are more issues involved firebase/firebase-tools-ui#332 so I'd suggest you run java on native Windows CMD if possible.)

I will try changing to 0.0.0.0 or 127.0.0.1 as suggested, I am using the native Windows CMD

[yuchenshi]

What variables do I need to setup?

There is only FIRESTORE_EMULATOR_HOST=localhost:8080 that needs to be set up. You may need to do something like:

set FIRESTORE_EMULATOR_HOST=localhost:8080
npm run my-test-script

Again it depends on your test setup, so it'll be great if you can share a snippet.

[olivierkrener]

I actually had an issue in my test suite that I have fixed now. So increasing the heap space to the maximum 2048m does not solve the issue, I still get an error

Aug 07, 2020 11:45:29 AM io.gapi.emulators.netty.HttpHandler$1 onError INFO: Exception when handling request: Java heap space

Fyi the command line I am running to start the emulator is

java -Xmx2048m -Duser.language=en -jar pathtoemulator\cloud-firestore-emulator-v1.11.5.jar --host localhost --port 8080 --rules pathtorules\firestore.rules node pathtoemulator\emulators\ui-v1.1.1\server.bundle.js

Anything else I could investigate or logs I could provide?

[yuchenshi]

Sorry, I think at this point there is simply not enough RAM to support this. We can definitely file a bug for making the coverage report generation more memory efficient on our side. But other than that, I'd suggest exporting coverage report more often (e.g. in the middle of the suite). You can also use a different project ID per test, if possible.

Note that test coverage can also be merged externally by manipulating the JSON object.

[olivierkrener]

I dont want to split my report as I want to see the overall coverage. Also, note that I am getting the same issue importing the JSON object only (so the memory issue is during the JSON object generation, not the report generation).

I will try to optimize my tests and limit the backend request meanwhile keeping the same coverage.

[yuchenshi]

I dont want to split my report as I want to see the overall coverage.

Thanks for providing the use case. We don't have a better answer for now, but just kindly note that you can export separately and write a couple lines of code to join those JSON files together. You won't lose any information that way and you'll still have a complete coverage report after joining.

[fractalf]

I'm also running into java.lang.OutOfMemoryError: Java heap space using cloud-firestore-emulator-v1.11.7.jar when trying to import production data into the firestore emulator. Would be nice if one could pass an argument upping the memory limit

[debug] [2020-08-26T08:59:33.294Z] Exception in thread "HTTP-Dispatcher"  {"metadata":{"emulator":{"name":"firestore"},"message":"Exception in thread \"HTTP-Dispatcher\" "}}
[debug] [2020-08-26T08:59:33.295Z] java.lang.OutOfMemoryError: Java heap space
 {"metadata":{"emulator":{"name":"firestore"},"message":"java.lang.OutOfMemoryError: Java heap space\n"}}
[debug] [2020-08-26T08:59:33.295Z] 	at java.base/java.util.HashMap$KeySet.iterator(HashMap.java:912)
	at java.base/java.util.HashSet.iterator(HashSet.java:173)
 {"metadata":{"emulator":{"name":"firestore"},"message":"\tat java.base/java.util.HashMap$KeySet.iterator(HashMap.java:912)\n\tat java.base/java.util.HashSet.iterator(HashSet.java:173)\n"}}
[debug] [2020-08-26T08:59:33.295Z] 	at java.base/sun.nio.ch.Util$2.iterator(Util.java:352)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:369)
	at java.base/java.lang.Thread.run(Thread.java:834)
 {"metadata":{"emulator":{"name":"firestore"},"message":"\tat java.base/sun.nio.ch.Util$2.iterator(Util.java:352)\n\tat jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:369)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}}
[debug] [2020-08-26T08:59:33.307Z] Aug 26, 2020 10:59:33 AM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
INFO: operation failed: The com.google.storage.onestore.v3.proto2api.OnestoreEntity$EntityProto message could not be parsed with the com.google.io.protocol.Proto2ParserAdapter parser.
 {"metadata":{"emulator":{"name":"firestore"},"message":"Aug 26, 2020 10:59:33 AM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError\nINFO: operation failed: The com.google.storage.onestore.v3.proto2api.OnestoreEntity$EntityProto message could not be parsed with the com.google.io.protocol.Proto2ParserAdapter parser.\n"}}
[debug] [2020-08-26T08:59:33.462Z] Aug 26, 2020 10:59:33 AM io.gapi.emulators.netty.HttpHandler$1 onError
INFO: Exception when handling request: UNKNOWN
 {"metadata":{"emulator":{"name":"firestore"},"message":"Aug 26, 2020 10:59:33 AM io.gapi.emulators.netty.HttpHandler$1 onError\nINFO: Exception when handling request: UNKNOWN\n"}}

[olivierkrener]

Increasing the heap size to 4Go solved my problem:

java -Xmx4g -Duser.language=en -jar pathtoemulator\cloud-firestore-emulator-v1.11.7.jar --host localhost --port 8080 --rules pathtorules\firestore.rules node pathtoemulator\emulators\ui-v1.1.1\server.bundle.js

Fyi, I can indeed see that during the report generation, the emulator process RAM consumption goes up to 2.8Go.

This issue can be closed from my point of view.

[yuchenshi]

Thanks, @olivierkrener! I'll repurpose this issue to track a better way to specify JVM flags in firebase emulators:start without having to dig out the jar.

[ghost]

Any update on this issue?

[olivierkrener]

Any update on this issue?

As I mention it can be fixed if you call the emulator directly specifying the heap size

[yuchenshi]

Thanks, @olivierkrener! Also for anyone interested: feel free to send us a Pull Request for adding a firebase.json configuration option for it, so that you can start emulators using Firebase CLI as usual.

[FocuzJS]

Bump, it seems there is a big need for this as our database is only 180MB and we are running into this issue!

[RayyanI]

Thanks, @olivierkrener! Also for anyone interested: feel free to send us a Pull Request for adding a firebase.json configuration option for it, so that you can start emulators using Firebase CLI as usual.

When will a solution for this be introduced?

[yuchenshi]

I've escalated this internally (b/179491197) but anybody can help to make this happen!

[arnaudbouchard]

Does anyone has a way to call the emulator directly specifying the heap size and importing data into Firestore (as you can do by adding the --import flag with firebase emulators:start)

[NikolajLeischner]

A workaround is to put the JVM flags into the environment variable JAVA_TOOL_OPTIONS, which is intended for cases where it is not easily possible to pass arguments to the JVM directly. At least OpenJDK, OpenJ9 and OracleJDK support this.

[samtstern]

@NikolajLeischner thanks for pointing that out! JAVA_TOOL_OPTIONS sounds like exactly what we want in this case. I was going to propose something like FIREBASE_EMULATORS_JAVA_OPTIONS but I think that would be redundant. And I do think an env var is cleaner than firebase.json as this feels like leaking an implementation detail (only half the emulators are Java).

@olivierkrener what do you think?

[olivierkrener]

@samtstern As I wrote I am calling the emulator directly via the java command line passing a custom heap size, so this is not an issue for me anymore

[yuchenshi]

IMO we can start with documenting JAVA_TOOL_OPTIONS in the emulator docs and we can reopen if that proves to be inadequate.

[ezmegy]

I'm not sure why but for me running the raw command

java -Xmx4g -Duser.language=en -jar /Users/peterm/.cache/firebase/emulators/cloud-firestore-emulator-v1.13.1.jar --host localhost --port 5050 --seed_from_export /Users/peterm/pathToMyBackupData/myBackupData.overall_export_metadata --rules /Users/peterm/pathToMyFirestoreRules/firestore.rules --functions_emulator localhost:5001 node /Users/peterm/.cache/firebase/emulators/ui-v1.6.0/server.bundle.js

didn't do it. It said the Dev App Server was running but just couldn't get the emulator UI to start working.

(Note1: tried "0.0.0.0" and "127.0.0.1" instead localhost as well. Note2: the port numbers above are reflected in my firebase.json file.)

But setting the java tool option to 4gigs in the environment variable did the trick:

export JAVA_TOOL_OPTIONS="-Xmx4g" (on mac)

So would definitely vote for adding this latter to the docs.

[yuchenshi]

@ezmegy tl;dr the jar contains only the Firestore Emulator while the Emulator UI is part of Firebase CLI. I'll see if we can get the env var documented soon.

[AverageHelper]

To future googlers: as of now, there is some documentation on setting JVM arguments in Firebase emulators.