Just for some queries: MDB_CORRUPTED: Located page was wrong type.

[sznowicki]

Describe the bug

Hello there.

After I migrated meilisearch to a new version (along with migration to a new disk from ext4 to lvm + LUKS with ex4 system) I noticed that for some queries it throws exception.

I believe I might have some files corrupted but I don't know how and if there's any way to fix them.

Any support highly appreciated ;)

Error message:

{"message":"Inside `.queries[0]`: internal: MDB_CORRUPTED: Located page was wrong type.","code":"internal","type":"internal","link":"https://docs.meilisearch.com/errors#internal"}

Screenshots Fine:

Broken:

Meilisearch version:

1.6.1

Additional context

Ubuntu, meili running on docker with mounted volume.

[sznowicki]

I thought about making a dump and recreate indexes but this also failed with also quite cryptic message:

Task {
  indexUid: null,
  status: 'failed',
  type: 'dumpCreation',
  uid: 2162947,
  details: { dumpUid: null },
  canceledBy: null,
  error: {
    message: 'internal: MDB_PAGE_NOTFOUND: Requested page not found.',
    code: 'internal',
    type: 'internal',
    link: 'https://docs.meilisearch.com/errors#internal'
  },
  duration: 'PT35.734438274S',
  startedAt: 2024-02-23T19:59:11.542Z,
  enqueuedAt: 2024-02-23T19:59:11.533Z,
  finishedAt: 2024-02-23T19:59:47.277Z
}

[Kerollmops]

Hey @sznowicki 👋

Thank you for the issue and report. The MDB_CORRUPTED kind of errors is clearly showing that the database is corrupt on disk.

It brings the question of what kind of volume did you mount on Docker? Is it a network disk (NTFS)?

Meilisearch, and more specifically LMDB (our internal key-value store), doesn't support network file systems. Always use a raw SSD or NVMe.

The best solution to avoid having to deal with this kind of issues is to use the cloud offering. We provide more features like analytics and auto-updates. We never saw any database corruption in the past.

[sznowicki]

It's RAID 1 setup with LVM and LUKS encryption. For the process it should be visible as a regular ext4 filesystem. I'm reindexing all from scratch at the moment and so far don't observe any problems.

Possibly something went wrong when I moved the data from one disk to another.

For future problems, is there any way to recover from such state using some LMDB processes?

[Kerollmops]

Possibly something went wrong when I moved the data from one disk to another.

Ho! Indeed, copying a file while it is being written to (indexing documents) is unsafe as the copy process is iterative (pages by pages) while data is being modified.

For future problems, is there any way to recover from such state using some LMDB processes?

There is maybe a feature in LMDB to read the pages from the previous transaction (before the one indexing new documents). But not sure it is available in LMDB 0.9...

[sznowicki]

I'm sure I've been moving data while meili process was not running though. I'll observe it with fresh index and report if it comes back. This time I'll make a proper dump ;)

[curquiza]

Hello @sznowicki did you get the issue again?

[sznowicki]

Hello @sznowicki did you get the issue again?

no, not so far and I actively push to the index pretty often, also did some dump import when I did version bump

[curquiza]

Ok, thank you for your feedback 🙏 I will close the issue. But feel free to reopen or open a new one if you encounter any problem