Firestore emulator in datastore mode should support --import and emulator:export

[Nathanmalnoury]

[REQUIRED] Environment info

firebase-tools: I'm using the gcloud CLI :

$ /usr/lib64/google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator --version
cloud-firestore-emulator 1.19.9

Platform:

Fedora Linux 40 (Workstation Edition)

[REQUIRED] Test case

https://github.com/Nathanmalnoury/repro-firestore-emulator-datastore-mode-persistence

[REQUIRED] Steps to reproduce

Included in test repo

[REQUIRED] Expected behavior

I would expect to see saved data in my configured "--export-on-exit" folder, to then be able to import them on the next start via --import-data.

[REQUIRED] Actual behavior

It stays empty when interrupting the emulator using Ctrl + C.

/Projects/test/building-an-app-3 (main ✔) gcloud emulators firestore start --database-mode datastore-mode   --export-on-exit ./saved_data  --host-port 0.0.0.0:8081 --project [PROJECT-ID] --verbosity debug 

DEBUG: Running [gcloud.emulators.firestore.start] with arguments: [--database-mode: "datastore-mode", --export-on-exit: "./saved_data", --host-port: "<googlecloudsdk.calliope.arg_parsers.HostPort object at 0x7f1313866310>", --project: "[PROJECT ID]", --verbosity: "debug"]
DEBUG: Found Cloud SDK root: /usr/lib64/google-cloud-sdk
Executing: /usr/lib64/google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator start --host=0.0.0.0 --port=8081 --database-mode=datastore-mode --export-on-exit=./saved_data
[firestore] Oct 29, 2024 11:28:12 AM com.google.cloud.datastore.emulator.firestore.websocket.WebSocketServer start
[firestore] INFO: Started WebSocket server on ws://0.0.0.0:37901
[firestore] API endpoint: http://0.0.0.0:8081
[firestore] If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:
[firestore] 
[firestore]    export FIRESTORE_EMULATOR_HOST=0.0.0.0:8081
[firestore] 
[firestore] If you are running a Firestore in Datastore Mode project, run:
[firestore] 
[firestore]    export DATASTORE_EMULATOR_HOST=0.0.0.0:8081
[firestore] 
[firestore] Note: Support for Datastore Mode is in preview. If you encounter any bugs please file at https://github.com/firebase/firebase-tools/issues.
[firestore] Dev App Server is now running.
[firestore] 
[firestore] Oct 29, 2024 11:28:51 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:51 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:53 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:53 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
^C

Command killed by keyboard interrupt

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[Nathanmalnoury]

To add some context to this issue, my company is based on ndb which is a google official client library for Google Cloud Datastore.

We've been using the legacy datastore emulator for local testing before deploying to pre-production and production projects. Data persistence between runs is crucial for our testing process, and I wasn't able to achieve with the firestore emulator. The core issue is that the 2.3.0 release of ndb (1st March 2024) rendered the ndb client incompatible with the legacy emulator (described in this issue). We find ourselves in an uncomfortable position where we can't update our ndb client because that would mean degrading our local setup to a point that would make it practically unusable.

I've spent some time reading the docs and trying to get the emulator to persist data, but I also may have missed something..

Thank you in advance for your time and help!

[aalej]

Hey @Nathanmalnoury, sorry to hear you encountered this issue and thanks for the thorough report. I appreciate you taking the time to put together an mcve! I was able to reproduce the behavior you mentioned, I'll raise this to our engineering team so that they can take a look.

As a temporary workaround, you could try using Import entities into the emulator and Export entities in the emulator to import/export data into/in the emulators.

[AlexisH]

Same issue, the data is not persisted, I don't see anything in the output dir. Was also previously using legacy datastore emulator, but I persisted the data in a different directory.

Ubuntu 22.04.5 LTS cloud-datastore-emulator 2.3.1 cloud-firestore-emulator 1.19.9

gcloud emulators firestore start --database-mode=datastore-mode --export-on-exit=/home/me/_firestore --host-port=127.0.0.1:8081

[joehan]

Hey all, quick question to help us debug this issue - does the exported data show up if you run firebase emulators:export outputDir in a separate terminal while the emulators are running?

[AlexisH]

I just installed firebase CLI for testing purpose. firebase emulators:export does not find the running emulator if I start it with gcloud emulators firestore start ... If I start the emulator via firebase emulators:start --only firestore --project foo then the export command finds it. If I start the emulator via firebase emulators:start --only firestore --project foo --export-on-exit outputdir then it correctly dumps to outputdir on exit. But using this emulator I can't get my python app to work I get Database mode not set to Datastore Mode. If using the gCloud CLI, please specify the --database-mode flag. How can I set this flag using the firebase CLI ?

[laluha]

Are there any progress on fixing this issue? .. Or should I stop using datastore-mode?

[drei01]

@joehan any update on this issue? The official docs say that persistence should work.

[vexdev]

@drei01 the official docs are wrong in that --export-on-exit does not seem to be working at all, however the import seems to be working for me. My workaround is to add a trap to perform the cleanup, I prepared a simple script which I run as an entrypoint in docker, but could work also standalone:

Notably provide the project name in CLOUDSDK_CORE_PROJECT, the port of the emulator in DS_PORT and the export folder in DS_EXPORT_PATH.

#!/bin/bash

cleanup() {
  echo "Saving state and cleaning up..."
  curl --fail --show-error --max-time 30 -X POST "http://localhost:$DS_PORT/emulator/v1/projects/$CLOUDSDK_CORE_PROJECT:export" \
    -H 'Content-Type: application/json' \
    -d "{\"database\":\"projects/$CLOUDSDK_CORE_PROJECT/databases/\", \"export_directory\":\"/$DS_EXPORT_PATH\"}"
  echo "Cleanup complete"
  exit 0
}

trap cleanup SIGTERM SIGINT SIGQUIT SIGHUP ERR

state_file=$(find "$DS_EXPORT_PATH" -exec readlink -e '{}' \; | grep export_metadata | sort | tail -n1)

if [ -z "$state_file" ]; then
  echo "No state file found in $DS_EXPORT_PATH. Running without import."
  gcloud emulators firestore start \
      --verbosity=debug \
      --host-port="0.0.0.0:$DS_PORT" \
      "--database-mode=datastore-mode" &
else
  echo "State file found: $state_file. Importing data."
  gcloud emulators firestore start \
      --verbosity=debug \
      --host-port="0.0.0.0:$DS_PORT" \
      "--database-mode=datastore-mode" \
      "--import-data=$state_file" &
fi

wait $!

[jkb016]

I am also facing same issue on windows 10 with following command

gcloud emulators firestore start --database-mode=datastore-mode --host-port=localhost:8081  --export-on-exit=D:\data

The export api from document reference https://cloud.google.com/datastore/docs/emulator works .

[anoteboom]

Is there any news on this bug? It is horrible as it is, that we are forced to move to the firebase emulator, with so many much loss of functionality compared to the datastore emulator. No stable data persistence. No useful index support.

We cannot upgrade to the latest version of the ndb library. We are stuck with 2.2.2, and it has started throwing deprecation warnings.

Why is there no option to save the database to disk, instead of doing this export-import stuff?