moonraker
moonraker copied to clipboard
Arksine
Moonraker's LMDB database files are architecture dependant
When moving the database folder from a 32bit ARMv7 SBC to a 64bit ARMv8 SBC, Moonraker fails to start with MDB_INVALID: File is not an LMDB file. This seems to be because Moonraker's database uses LMDB, which has architecture dependant database files. I doubt many people will ever run into this issue, but it'd be good to have some sort of automatic architecture conversion or a script to do it.
I did run. Moving from rock64 into Raspberry PI Zero 2 W.
This is becomes less of the problem if you move from 64 bit to 32 bit, but:
- It requires compiling
LMDBwithMDB_VL32 - With this flag all databases created are forward compatible (64 bit by default)
- WIth this flag all current 32 bit databases are broken
- This flag is not yet released, as it is only
masterfor now https://github.com/LMDB/lmdb/search?q=MDB_VL32 - This flag is not translated into https://github.com/jnwatson/py-lmdb/blob/master/lib/mdb.c
Maybe there are better alternatives?
It appears that it should be possible to do mdb_dump on source platform to perform mdb_load on target platform.
Right, as mentioned lmdb-tools provides a facility for transferring databases between architectures with mdb_save and mdb_load. The pylmdb package also provides one that I have not tested.
Moving to another DB implementation isn't something I would consider. IMO its benefits outweigh the minor inconvenience of being architecture dependent. I do agree that adding scripts to import and export the database is a good idea.
Yup. I managed to convert everything with:
apt-get install lmdb-utils
ssh [email protected] mdb_dump -a /opt/klipper-data/moonraker_database | mdb_load /opt/klipper-data/moonraker_database
how to move if from 32 bit to 64 bit system? can you please also describe this?
//solved
mdb_dump -f backup.txt -a .moonraker_database mdb_load -f backup.txt -s -T ~/.moonraker_database
In my case I had to run mdb_dump on a 32 bit architecture OS to dump the file; the rest was exactly like the others have said.
Thank you for this. I needed it to be able to keep my database while going from a x64 MainsailOS image to x86.
The mdb_dump worked as expected, although I did modify the path for MainsailOS and current folders
apt-get install lmdb-utils
mdb_dump -f backup.txt -a ~/printer_data/database
After reinstalling MainsailOS x86 and copying over the backup.txt, I did have to remove the current database file, since I had also restored it from my previous image and I couldn't restore into it
rm ~/printer_data/database/* -rf
mdb_load -f backup.txt -s -T ~/printer_data/database
Well I am having the worst time trying to do this. My printer sat idle for months and when I restarted it I upgraded mainsail and other things and it all broke everywhere so I started over. After trying to copy my mainsail database over to the new Pi install this isn't working for me:
pi@mainsailos:~ $ mdb_dump -f backup.txt -a .moonraker_database
mdb_env_open failed, error -30793 MDB_INVALID: File is not an LMDB file
When you reinstalled, did you happen to change from an x86 install to x64 or the other way around?
From: Dennis @.> Sent: Saturday, February 10, 2024 5:32:41 PM To: Arksine/moonraker @.> Cc: MallocArray @.>; Comment @.> Subject: Re: [Arksine/moonraker] Moonraker's LMDB database files are architecture dependant (#192)
Well I am having the worst time trying to do this. My printer sat idle for months and when I restarted it I upgraded mainsail and other things and it all broke everywhere so I started over. After trying to copy my mainsail database over to the new Pi install this isn't working for me:
@.***:~ $ mdb_dump -f backup.txt -a .moonraker_database mdb_env_open failed, error -30793 MDB_INVALID: File is not an LMDB file
— Reply to this email directly, view it on GitHubhttps://github.com/Arksine/moonraker/issues/192#issuecomment-1937352787, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACHQZJBCKFY5ZP6B7TNU73DYS77RTAVCNFSM5AKS3DYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTG4ZTKMRXHA3Q. You are receiving this because you commented.Message ID: @.***>
OK, sorted it out based on a YouTube video I watched.
The architecture was exactly the same as it was the same Pi I was using.
Here's what ended up working for me:
- Replace the SD card in the Pi with the old one and boot up the system.
- My problems were that mainsail didn't start due to corrupt files / blocks but the system itself booted.
- SSH'd in and ran the mdb_dump command and it worked.
- FTP'd the file to another linux server and shutdown the Pi.
- Replaced the SD card with the new one, started things up, stopped moonraker and did the mdb_load command to restore.
- Things are good - my history is there with the restored DB.
I posted this in moonraker channel on Fluidd Discord, but there does not seem to be much traffic there and I didn't want to ping Arksine directly:
I came across what appears to be a Moonraker issue while restoring my 32 bit Bullseye Klipper installation backup to new 64 bit Bookworm, both on the same Raspberry Pi but with a new SD card. The original Moonraker database was backed-up and restored using the backup-database.sh and restore-database.sh scripts. Upon startup of the new installation, Moonraker appears to modify the restored database in some way that removes most, but not all "Last printed" data from the Fluidd displayed print jobs. At first all the "Last printed" data is available while Fluidd displays an animated horizontal blue line in the Jobs frame. After some time the animation stops and almost all "Last printed" data is replaced by "-". Curiously the print history data is unaffected and continues to correctly show the complete print history.
And to illustrate what I mean, here is a sample Fluidd display immediately after Moonraker restart following restore-database.sh or mdb_load:
Which then becomes this after things settle down, noting that the "Last printed" dates disappear:
Meanwhile data in the print history records remains unaffected:
@ReXT3D This is likely the correct behavior. The last printed field is offered in the file metadata. That is, it applies to the exact file, if the file is modified in any way (such as copied) the metadata is reparsed and the fields that provide the last print time and last job id are reset and reported as null.
When you moved your installation Moonraker detected that the modified time for each gcode file has changed, so it reparsed the metadata. This explains why the removal occurs after Moonraker 'settles'.
Fluidd likely could report its Last Printed field based on the history and the file name, but if the file were overwritten or changed it wouldn't necessarily be accurate.
When you moved your installation Moonraker detected that the modified time for each gcode file has changed, so it reparsed the metadata. This explains why the removal occurs after Moonraker 'settles'.
Eureka! Thanks for the explanation @Arksine!
On unix & like systems I have always used tar for backups and I expect it to preserve the timestamps correctly by default - after all it's a "tape archiver" that was designed specifically for making identical file backups! For reasons only clear to those responsible, tar on Linux defaults to storing time stamps with one second granularity even though the file system stores time stamps in much higher resolution. So my "identical" backups were not so identical after all!
To address this issue, in case anyone else comes across it and is equally OCD, use the following commands on Linux based host in the printer_data directory:
To backup the gcodes directory: tar --format=posix -cvpf gcodes.tar gcodes
To restore the gcodes directory: tar --atime-preserve -xvpf gcodes.tar
The above tar switches will preserve nanosecond precision for file timestamps and prevent Moonraker from reparsing the metadata.
For anyone else with this issue on a raspberry pi, you can install the 32bit version of mdb_dump, dump the database, install the 64bit version of mdb_dump, and load it back, all from one 64bit OS:
sudo dpkg --add-architecture armhf
sudo apt install lmdb-utils:armhf
mdb_dump -f backup.txt -a ~/path/to/old/database
sudo apt install lmdb-utils
rm ~/printer_data/database/* -rf
mdb_load -f backup.txt -s -T ~/printer_data/database
Since moonraker now uses sqlite instead https://github.com/Arksine/moonraker/commit/80c762074ff7100533f35f26131d39b9b749cc88 this shouldn't be an issue going forward. It's probably a good idea to note something in the docs for people restoring from backups, or otherwise interacting with older versions of moonraker. IIUC, moonraker will automatically migrate the DB to sqlite, I assume this is a one-way operation, so that commit is a point-of-no-return.
Indeed, this should be resolved with the move to sqlite.
FWIW, when Moonraker performs the conversion it does not remove the lmdb database, so a user can revert if something goes wrong. For example, some users are running devices stuck on Ubuntu 18.04, which ships with a version of Sqlite that doesn't support the UPSERT functionality used by Moonraker. These users were able to revert back to the commit prior to the Sqlite change and continue using Moonraker on their machines.