Lost bookmarks since upgrade to 1.5.0

[KillianKemps]

Hello,

I upgraded shiori from 1.0-1 to 1.5.0-2 on my computer on Arch Linux. I couldn't login with my previous credentials so I logged in with the default as described here. However, the list of bookmarks was empty. I recreated an account with the same username, but a different password and still no bookmarks were listed.

When I check the file at ~/.local/share/shiori/shiori.db I see the data is still here. I did a backup and then tried to add a new bookmark with the new account. The file at ~/.local/share/shiori/shiori.db has not been modified, but the new bookmark seems to be saved somewhere.

It looks like the storage location has changed, but if I read the documentation, the default data directory seems to be this.

Do you have an idea of the reason of the lost bookmarks?

[fmartingr]

Hey @KillianKemps, can you run find . -name shiori.db 2> /dev/null and share the output to check were the shiori.db may have been created? Remember that if you run it in portable mode (--portable) the database will be created where the shiori executable is, alternatively it should use the SHIORI_DIR environment variable or the data path you shared (which may have changed in the latest commits, I need to check that).

[kevinfiol]

I had a similar experience, although in my case, all bookmarks were lost after I attempted to upgrade to 1.5.1 from a build from master (commit a950bfec4c75eadbe5c1a03302ba9875a8236d45). Simply replacing the shiori binary, and attempting to run shiori serve failed with Failed to create data dir: mkdir /home/kevin/.local/share/shiori/shiori.db: not a directory. At that point, I checked, and indeed the .db file was still there, but unfortunately, my account and all DB data was wiped.

I'm not familiar with the codebase, so I'm not certain what might have happened here.

[fmartingr]

I had a similar experience, although in my case, all bookmarks were lost after I attempted to upgrade to 1.5.1 from a build from master (commit a950bfe). Simply replacing the shiori binary, and attempting to run shiori serve failed with Failed to create data dir: mkdir /home/kevin/.local/share/shiori/shiori.db: not a directory. At that point, I checked, and indeed the .db file was still there, but unfortunately, my account and all DB data was wiped.

I'm not familiar with the codebase, so I'm not certain what might have happened here.

Hey @kevinfiol. First of all, apologies for what happened! I don't understand what could've happened. I update my own shiori in each feature merged to master (but not released) to check that everything goes smoothly, but clearly I missed something.

First of all, can you confirm that you lost the database? I made a quick test in an empty machine fist executing 1.5.0, creating an user and adding a bookmark, then running 1.5.1 and got the error you mention:

Failed to create data dir: mkdir /home/fmartingr/.local/share/shiori/shiori.db: not a directory

But if I stop 1.5.1 and run 1.5.0 again the database is still there. Can you check that? I believe that whatever happend hasn't 100% destructive (or I hope so).

I will investigate futher tonight once I have some free time available.

[kevinfiol]

Hi @fmartingr thanks for looking into this. I did confirm that my data was wiped by inspecting my .db files using multiple SQLite clients (including https://sqlitebrowser.org/), but let me know if there's something else I could do. I also tried reverting back to my older shiori binary to see if I could still login & access my bookmarks, but I could not even log in (DB still showed that the accounts and bookmarks tables had 0 rows each).

For more context, my initial binary was an ARMv7 build compiled from the repo at the commit I mentioned above (a950bfec4c75eadbe5c1a03302ba9875a8236d45) and I run Shiori on Raspbian. Let me know if there's anything else I could do to help out.

[soulteary]

Perhaps a data upgrade detection tool should be made to prevent other users from encountering similar problems.

[fmartingr]

Hi @fmartingr thanks for looking into this. I did confirm that my data was wiped by inspecting my .db files using multiple SQLite clients (including https://sqlitebrowser.org/), but let me know if there's something else I could do. I also tried reverting back to my older shiori binary to see if I could still login & access my bookmarks, but I could not even log in (DB still showed that the accounts and bookmarks tables had 0 rows each).

For more context, my initial binary was an ARMv7 build compiled from the repo at the commit I mentioned above (a950bfe) and I run Shiori on Raspbian. Let me know if there's anything else I could do to help out.

I've made some tests migrating manually between versions using default paths, SHIORI_DIR and in any case the database should've been wiped by shiori. The error you've got means Shiori couldn't create the folder where the database resides, so it just fails and stops there. Did at any point give you any other error messages?

Perhaps a data upgrade detection tool should be made to prevent other users from encountering similar problems.

I totally agree, but this is only for the database side since the issue here was a bug where shiori was looking for the database where it shouldn't (fixed in #378). The fts4 to fts5 migration is being worked on #379 as a "workaround" until we have proper migrations and/or an ORM to work with.

[soulteary]

When I was cleaning up the notification, I saw that the previous pr had a maintenance reply, and found that this project has been resurrected, which is a good thing, congratulations 🎉

I'm thinking about making some simple feature commits and bugfixes for this project.

However, I don't see the new milestones for this project, and the merge criteria for PR. In order to avoid invalid pr again, maybe you can give me some guidance, thank you. @fmartingr

[fmartingr]

Ok, I've done a few things:

• First, a notice added in v1.5.1 which points to a required migration for SQLIte users, ~~I will add an announcement as well.~~ Announcement here
• I've released v1.5.2 with the path fix for the Shiori sqlite dir.
• I've created a tool to migrate an SQLite database from FTS4 to FTS5 here (explained in the v1.5.1)

Regarding the bug you experience, I've failed to reproduce it, and looking at the part of the code that gives the error, shiori shouldn't been able to create an empty database since it didn't know the path at that point. The only error I experienced regarding the 1.5.0 to 1.5.1 upgrade was the FTS module error (#377), which I hope its solved already. The only thing I haven't tested already is the build on an ARM device because I do not have any to spare at the moment.

@soulteary Hey! If you don't mind, we can move this conversation to the IRC (I'm online 24/7 through a relay so I can have async conversations in the channel) or open a discussion so we can chat in there. There's no roadmap because there's too much to cover at the moment. There are some features I want to address sooner than later, but right now I'm focusing on not screwing more releases :sweat:

[KillianKemps]

Hey @KillianKemps, can you run find . -name shiori.db 2> /dev/null and share the output to check were the shiori.db may have been created? Remember that if you run it in portable mode (--portable) the database will be created where the shiori executable is, alternatively it should use the SHIORI_DIR environment variable or the data path you shared (which may have changed in the latest commits, I need to check that).

Hello, I'm answering only now, but here is the result of the command:

% find . -name shiori.db 2> /dev/null
./.local/share/shiori/shiori.db

Shiori is run with this simple command: /usr/bin/shiori serve -p 8777

Checking the database with SQLite, the 800+ bookmarks are still here, but I only see one bookmark when I connect to my account. I understand that I lost my previous account and that my new account, even though I recreated it with the same name (it is still has the ID 1 in the account table in SQLite), isn't linked to the bookmarks anymore. Looking at the tables in the database, I don't understand at which moment the bookmarks are linked to a specific account? Are all the bookmarks shared with all the accounts?

I precise that I'm still running version 1.5.0-2 of Shiori and I understand I'm not concerned by the SQLite migration needed for 1.5.1.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity for quite some time. It will be closed if no further activity occurs. Thank you for your contributions.

[fmartingr]

Looking at the tables in the database, I don't understand at which moment the bookmarks are linked to a specific account? Are all the bookmarks shared with all the accounts?

Right now every bookmark can be seen by any user, guests can see and admins can manage bookmarks.

If you have the correct database file, make a backup of that entire folder (/.local/share/shiori/)try to run shiori using SHIORI_DIR=$HOME/.local/share/shiori and check if you have everything in there, if you only see one bookmark when running serve is because the data directory is most likely somewhere else.

[KillianKemps]

I did the backup of the /.local/share/shiori/ folder and ran an instance of Shiori:

SHIORI_DIR=$HOME/.local/share/shiori shiori serve`

All the bookmarks are again available with this instance!

I then ran the original Shiori instance which used to be launched by Systemd and I still only had access to one bookmark. I looked into the systemd service file and here is the configuration:

[Service]
ExecStart=/usr/bin/shiori serve -p 8777
ExecStop=pkill shiori
Environment=ENV_SHIORI_DIR=/home/killian/.local/share/shiori

It looks like the ENV_SHIORI_DIR environment variable changed its name through the versions to SHIORI_DIR? Or was it already a typo from the beginning?

[KillianKemps]

Yes in fact the environment variable indeed changed as noted in the docs.

Reading again the issue from the beginning, you may have already mentioned it in your first comment, but I didn't understand at that time that the variable name changed.

Hey @KillianKemps, can you run find . -name shiori.db 2> /dev/null and share the output to check were the shiori.db may have been created? Remember that if you run it in portable mode (--portable) the database will be created where the shiori executable is, alternatively it should use the SHIORI_DIR environment variable or the data path you shared (which may have changed in the latest commits, I need to check that).

Anyway, renaming the environment variable fixed the issue. Thanks for you help!