Meili API key errors after backup restore

[pleonex]

While testing my container deployment setup, I've found that restoring a backup is causing issues with MeiliSearch, and it fails to show the trails.

However, it works if I back up and restore the meili-data volume too (full /meili_data/data.ms folder). Should this be the expected process?

Reproduction steps

1. Create a new instance from the main Docker compose
2. Create the admin user in PocketBase (from the logs URL)
3. Create a new Wanderer user from PocketBase user tab (marking verified as true)
4. Login in Wanderer with the new user and upload a trail (in my case it's a 22km, I can share it if needed)
5. Create a backup from PocketBase admin panel and download it
6. Stop the docker containers (docker compose down -v) and remove the data directory
7. Start the containers again, but this time from the PocketBase initial user creation page (before creating the admin user), restore the backup.
8. Try to go into the /trails page -> it doesn't load and several issues are logged:

wanderer-search  | 2025-07-15T10:28:55.032874Z  WARN HTTP request{method=POST host="search:7700" route=/indexes/trails/search query_parameters= user_agent=node status_code=403 error=The provided API key is invalid.}: tracing_actix_web::middleware: Error encountered while processing the incoming HTTP request: ResponseError { code: 403, message: "The provided API key is invalid.", error_code: "invalid_api_key", error_type: "auth", error_link: "https://docs.meilisearch.com/errors#invalid_api_key" }
wanderer-search  | 2025-07-15T10:28:55.033108Z  INFO HTTP request{method=POST host="search:7700" route=/indexes/trails/search query_parameters= user_agent=node status_code=403 error=The provided API key is invalid.}: meilisearch: close time.busy=1.79ms time.idle=1.06ms
wanderer-web     | MeiliSearchApiError: The provided API key is invalid.
wanderer-web     |     at /app/node_modules/meilisearch/dist/bundles/meilisearch.cjs.js:378:19
wanderer-web     |     at Generator.next (<anonymous>)
wanderer-web     |     at fulfilled (/app/node_modules/meilisearch/dist/bundles/meilisearch.cjs.js:298:58)
wanderer-web     |     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
wanderer-web     |   code: 'invalid_api_key',
wanderer-web     |   type: 'auth',
wanderer-web     |   link: 'https://docs.meilisearch.com/errors#invalid_api_key',
wanderer-web     |   httpStatus: 403
wanderer-web     | }

In some of these tests, when I copy the files before the first container start, I also see: wanderer-search | 2025-07-15T09:56:56.751420Z ERROR index_scheduler: Batch failed Index ``trails`` not found.

I also tried restoring the backup from local files into the container:

1. docker compose create db
2. docker compose cp --archive backup/pb_data db:/

Or performing the backup by copying the /pb_data files when the container was running after the initial setup (also tried with the container stopped). Also tried running sqlite3 /pb_data/data.db ".backup /pb_data/data.db.bk" and sqlite3 /pb_data/auxiliary.db ".backup /pb_data/auxiliary.db.bk" (in case there were some transaction issues). But same issue always.

Looking at other similar issues, my guess is that the Meili index is regenerated with some issues. I tried running migrations up and down randomly, but it didn't help. I also tried logout and login but nothing.

[pleonex]

Update on my side. Just to be on the safe side, I decided to do these additional steps in my backups:

• Stop MeiliSearch and PocketBase containers before performing backups to be in a consistent state (Meili should pick up remaining tasks)
• Back up volume with the MeiliSearch data (/meili_data/data.ms).
• Configure MeiliSearch to perform a daily snapshot by setting the env vars: MEILI_SCHEDULE_SNAPSHOT=86400 and MEILI_SNAPSHOT_DIR=/meili_data/snapshots/
• Back up (new) volume with the MeiliSearch snapshots (/meili_data/snapshots/)
• Use Sqlite3 .backup command to back up PocketBase database. I ran inside the container: apk add sqlite && sqlite3 /pb_data/data.db ".backup /pb_data/data.db.bk" && sqlite3 /pb_data/auxiliary.db ".backup /pb_data/auxiliary.db.bk"

I use Offen volume backups to automatize the backup, if anyone is curious, my Docker compose files are here.

I guess for this ticket, we would need to either:

• Confirm if we should also backup the Meili database -> update doc
• If it's not necessary as the Wanderer documentation says -> investigate why it's failing or if we need to perform any other step to trigger the creation.

[Ra72xx]

As I saw in the Dockerfile that Wanderer nowadays does not use a custom Meilisearch container image but a standard one, I tried to switch to this, too, including update to the newest version. Reading the backup/restore chapter, I got the impression that this should be relatively painless, as for me it looked as if Meilisearch stuff would simply rebuild itself. So I did not really take precautions. But I also ran into this api key problem. I was not able to update to the latest Meilisearch image as the api key would no longer work. I really did not understand this, as also the Meilisearch documentation claims that changing the master api key is as simple as changing the corresponding env variable. So as Wanderer and Meilisearch use the same env variables, everything should have sorted itself out just fine. Which did not work for me. I had to revert to a backup and pinning the Meilisearch version to 1.11, using the same database files as before. I was not able to create a new Meilisearch database. Being not really a specialist in Meilisearch I got the impression that the documentation should be better here. The api key seems not to be something that can easily be changed.

[RobertClarke64]

This is affecting me too. I tried to upgrade meilisearch, and stupidly ended up just deleting its data as I thought it would just rebuild. But now I'm getting these invalid API key errors, even after downgrading back to meilisearch 1.11.3.

Now I can't use wanderer at all :(

[RobertClarke64]

I did notice that if I went to the meilisearch web UI and authenticated with the master key, I was able to see the trails that had populated back in. However on the wanderer-web UI, I was getting 403 (invalid api key) errors from the search API endpoint.

Is there a difference in the way the master key is fetched from environment variables when writing to meilisearch, and when querying it?

[pleonex]

Could it be related to this issue in Meili? https://github.com/meilisearch/meilisearch/issues/3902

It seems the workaround was to set the master key as a parameter in the service command.

I think Meili may have some issues reading some environment variables. I am having also issues with MEILI_SCHEDULE_SNAPSHOT that seems to be ignored as well.

[Ra72xx]

Could it be related to this issue in Meili? meilisearch/meilisearch#3902

This bug report was closed over 2 years ago, do you think it is still valid? However, I also noticed the thing with the quotes: If I pass the master key in an environment file, as I did initially, the quotes are part of the master key. As I did not change anything here, this should not be the problem in this case.

[Ra72xx]

This is affecting me too. I tried to upgrade meilisearch, and stupidly ended up just deleting its data as I thought it would just rebuild. But now I'm getting these invalid API key errors, even after downgrading back to meilisearch 1.11.3.

Now I can't use wanderer at all :(

I had to restore a nightly snapshot of meilisearch I fortunately had made in order to have access again to my tracks. @Flomp , this is a problem that can result in data loss when simply adhering to the instructions, I think this should be handled somehow.

[RobertClarke64]

Could it be related to this issue in Meili? meilisearch/meilisearch#3902

It seems the workaround was to set the master key as a parameter in the service command.

I think Meili may have some issues reading some environment variables. I am having also issues with MEILI_SCHEDULE_SNAPSHOT that seems to be ignored as well.

I don't think this is the problem. I tried adding that argument to the meilisearch command, but I have the same problem. I think this is a problem with the wanderer-web container. I noticed that the wanderer-db container is not showing any errors connecting to meilisearch (but if I changed the master key and don't update the db, it does). I'm still getting invalid api key errors from the wanderer-web pod though.

It is successfully populating trails into meilisearch (I think the wanderer-db container is doing this?), but wanderer-web isn't able to access any of them.

[RobertClarke64]

Something I've noticed is that I am able to view trails when logged out (my trails are public) or when logged in with another user. But I get the invalid API token errors when I log in with my normal user.

[RobertClarke64]

Okay I seem to have found the fix.

When a user is created, it seems like a meilisearch tenant token is created and stored in the database (see token column in image below). When the meilisearch data is deleted, this token is no longer valid. I'm not sure exactly when this token gets updated, perhaps only on new user creation. But we need a new valid token.

So to fix this, I created a new user in wanderer, then copied their token into my old user by directly modifying the database. I could then log in with my old user again and see all the trails.

[RobertClarke64]

@Flomp I think we need some way for this meilisearch tenant token to be regenerated.

[Ra72xx]

Any update on this? Currently I try not to touch my wanderer/meili installation, as I'm afraid I'll loose my tracks, but as of now it looks if there is not even a working backup solution apart from snapshotting the filesystem - as soon as something with meili changes, you might lose your wanderer information, too?

[Flomp]

Not yet, sorry. @RobertClarke64 is absolutely correct that the problems are the user tokens that likely need to be regenerated. I'll still be very busy through November. I hope I can fix this by early December with priority.