Datastore UI is not working with datastore in firestore mode

[sebastian-filip]

I am trying to make the UI work with datastore in firestore mode. This is my docker-compose.yaml

version: "3.9"
services:
  # DSAdmin container
  dsadmin:
    image: "ghcr.io/remko/dsadmin:latest"
    depends_on:
      - datastore
    ports:
      - "3334:8080"
    environment:
      DATASTORE_PROJECT_ID: "project-dev"
      DATASTORE_EMULATOR_HOST: "datastore:3333"

  # Datastore Emulator container
  datastore:
    image: "gcr.io/google.com/cloudsdktool/google-cloud-cli:latest"
    volumes:
      - datastore_data:/opt/datastore/data
    ports:
      - "3333:3333"
    command: [
      "gcloud", "emulators", "firestore", "start","--database-mode=datastore-mode", "--host-port=0.0.0.0:3333"
    ]
    environment:
      CLOUDSDK_CORE_PROJECT: "project-dev"

volumes:
  datastore_data:

But for some reason I keep on getting this Error.

What can I do about it?

[remko]

Good question. I have not tried the firestore emulator yet, I always use the datastore emulator.

From your error message, it sounds like it doesn't support the (metadata) queries that dsadmin does to list the namespaces and kinds available in the database. I would have to check whether there is a replacement for this in the firestore emulator.

[jmhobbs]

Bumped into this as well, it looks like it is in fact a limitation of the Firestore emulator in datastore mode,

[firestore] Jun 25, 2024 12:55:45 AM io.gapi.emulators.netty.HttpHandler$1 onError
[firestore] INFO: Exception when handling request: INVALID_ARGUMENT: Kind queries are not supported in the Datastore Mode emulator.

I could not find any docs stating this limitation though, nor any tickets on issuetracker.google.com.

I switched to the datastore emulator with the --use-firestore-in-datastore-mode flag on and it's working for my use case.

[sebastian-filip]

Thank you guys for your feedback.

The only issue I have is that I keep on getting the same error Filter has 4 properties, expected 1 whenever I try to use the IN filter. It seems that with firestore the issue is gone, and IN filtering works as expected.

This is also the solution Google is recommending for filtering.

@jmhobbs I tried now to use --use-firestore-in-datastore-mode such as:

DSAdmin container
dsadmin:
image: "ghcr.io/remko/dsadmin:latest"
depends_on:
- datastore
ports:
- "3334:8080"
environment:
DATASTORE_PROJECT_ID: "project-dev"
DATASTORE_EMULATOR_HOST: "datastore:3333"

Datastore Emulator container
datastore:
image: "gcr.io/google.com/cloudsdktool/google-cloud-cli:latest"
volumes:
- datastore_data:/opt/datastore/data
ports:
- "3333:3333"
command: [
"gcloud", "--quiet", "beta", "emulators", "datastore", "start", "--use-firestore-in-datastore-mode",
      "--no-store-on-disk", "--data-dir=/opt/datastore/data", "--host-port=0.0.0.0:3333"
]
environment:
CLOUDSDK_CORE_PROJECT: "project-dev"

volumes:
datastore_data:

And I keep getting the same error for IN filtering.

Does this work for you, or was this happening for you as well? How do you guys manage to do IN, !=, and NOT-IN queries?

[mattmelgard-qz]

Thank you guys for your feedback.

The only issue I have is that I keep on getting the same error Filter has 4 properties, expected 1 whenever I try to use the IN filter. It seems that with firestore the issue is gone, and IN filtering works as expected.

This is also the solution Google is recommending for filtering.

@jmhobbs I tried now to use --use-firestore-in-datastore-mode such as:

DSAdmin container
dsadmin:
image: "ghcr.io/remko/dsadmin:latest"
depends_on:
- datastore
ports:
- "3334:8080"
environment:
DATASTORE_PROJECT_ID: "project-dev"
DATASTORE_EMULATOR_HOST: "datastore:3333"

Datastore Emulator container
datastore:
image: "gcr.io/google.com/cloudsdktool/google-cloud-cli:latest"
volumes:
- datastore_data:/opt/datastore/data
ports:
- "3333:3333"
command: [
"gcloud", "--quiet", "beta", "emulators", "datastore", "start", "--use-firestore-in-datastore-mode",
      "--no-store-on-disk", "--data-dir=/opt/datastore/data", "--host-port=0.0.0.0:3333"
]
environment:
CLOUDSDK_CORE_PROJECT: "project-dev"

volumes:
datastore_data:

And I keep getting the same error for IN filtering.

Does this work for you, or was this happening for you as well? How do you guys manage to do IN, !=, and NOT-IN queries?

This is my problem - I can only get IN, NOT_IN and other filters to work when using the firestore emulator in datastore mode... But then namespaces don't work 🙃. Really awesome work as usual on Google's part with their crappy emulators...

[vexdev]

@remko Just a suggestion, apparently while not supporting __kind__ queries, firestore in datastore mode seems to support querying without kind, such as SELECT *. Obviously grossly inefficient to discover kinds like this, but it could be a workaround while waiting for better alternatives.

[remko]

@vexdev Interesting, thanks!

[drehelis]

@sebastian-filip I'm the author of https://github.com/drehelis/gcp-emulator-ui and I've just recently pushed support for Firestore emulator. LMK if it works for you. I'd be happy to hear feedbacks and suggestions.

[sebastian-filip]

Thx a lot for the support @drehelis but I am more interested in datastore emulator which I saw that is coming soon. Please let us know when that is workin, I am ready to test it as soon as it comes out. Cheers!

[drehelis]

@sebastian-filip FYI, just pushed datastore-mode support. ⭐

I obviously couldn't test every scenario, hope you can raise an issue if you stumble upon bugs etc.