Errors when importing many bookmarks (Server Errors 500, 405 and more)

[gingerbeardman]

Hi,

Thanks for this project!

I just set it up using docker and imported my ~4100 bookmarks from Pinboard.

• On import there was no success message, clicking the linkding logo to get away from that page resulted in server error 500.
• Clicking a tag resulted in a server error 500, clicking the same tag again now there is no error

Attached is a log from the docker container that shows the 500 errors.

Can I find out any information to help you diagnose/solve?

linkding.csv

[sissbruecker]

The import is definitely not very fast, and importing such a large number of bookmarks usually takes a while. According to the logs it seems you navigated to the homepage 10secs after starting the import, that seems too fast to import that amount of bookmarks.

On import there was no success message

Unfortunately there is no visual feedback while the import is running, but you can take a look at whether your browser is showing a loading indicator. After the import finishes, it should reload the page and display a message below the file upload field:

The errors kind of sound like the import was not completed, though I can't explain why that would result in the behavior you described. I would suggest to run the import and wait until you see the message above. If you are running into timeouts, take a look at this option and see if it improves things.

[gingerbeardman]

Interesting! For comparison Shaarli imported ~4100 almost instantly, a second or two.

I guess I need to check if all my bookmarks have imported. Is there anywhere i can see the total number of bookmarks?

Ah, 6 pages of 30 bookmarks means only 180 imported in 10 seconds!?

Hmm. So over 4 minutes to import all my bookmarks... 😧

[gingerbeardman]

Q: if I do the import again will it overwrite all the bookmarks that have already been imported?

[sissbruecker]

I guess I need to check if all my bookmarks have imported. Is there anywhere i can see the total number of bookmarks?

You can check the admin interface at /admin to get an overview of all the data in your linkding instance.

Q: if I do the import again will it overwrite all the bookmarks that have already been imported?

It should update existing bookmarks, using the URL for checking identity. It should definitely not create duplictates.

[gingerbeardman]

Excellent, I'll try again now.

Thanks for the quick support!

[gingerbeardman]

Doing a new import and waiting: Error 405 not allowed

Trying to get to /admin/bookmarks/bookmark/: Error 500

😕

[gingerbeardman]

I have increased LD_REQUEST_TIMEOUT to 300 (5 minutes) fingers crossed for third time lucky

Why is import so slow?

edit: still Error 405 after ~60 seconds.

[gingerbeardman]

It only imports about ~200 bookmarks each try.

4100 means I will have to do import many times.

I think I'll give up for today.

[gingerbeardman]

In console I'm seeing django.db.utils.OperationalError: database is locked

Currently up to 1800 bookmarks, but to be honest I worry about the integrity of the import so far.

edit: also seeing many

2022-02-22 13:58:41,stdout,Traceback (most recent call last):  2022-02-22 13:58:41,stdout,ERROR:root:Error importing bookmark: NetscapeBookmark(href='http://goldenageofgames.com/winhanafuda/' | title='Classic Personal Computer (... |   | ```

[gingerbeardman]

I've deleted the database and started again.

[gingerbeardman]

OK, no change. Giving up.

Please keep me informed if there are improvements to the import process. Many thanks.

[gingerbeardman]

If you'd like me to send you my bookmarks file privately, I'm happy to do so.

I'll strip the private links from it before hand.

[gingerbeardman]

I went away to do other stuff and thought I'd check on things... 4166 bookmarks imported.

So it seems the import has continued even with the 500/405/database/etc errors.

Pinboard listed 4170 bookmarks so perhaps it's complete?

But what about the missing 4 bookmarks?

[gingerbeardman]

OK! A little bit of text processing and diffing later, here are the missing 4 bookmarks:

<DT><A HREF="http://infoseek_rip.g.ribbon.to/crspalm.hp.infoseek.co.jp/" ADD_DATE="1554076853" PRIVATE="0" TOREAD="0" TAGS="hanafuda,palmos,japanese">Hiroki Takahashi : Palm/CLIE</A>

http://infoseek_rip.g.ribbon.to/crspalm.hp.infoseek.co.jp/

<DT><A HREF="http://a_pollett.tripod.com/mayumihf.htm" ADD_DATE="1549816635" PRIVATE="0" TOREAD="0" TAGS="hanafuda,rules">Mayumi-no Hanafuda · Mayumi's Hanafuda</A>

http://a_pollett.tripod.com/mayumihf.htm

<DT><A HREF="http://Guardian%20April%20Fools" ADD_DATE="1364908247" PRIVATE="0" TOREAD="0" TAGS="http://www.scribd.com/mobile/doc/87368327">San Serriffe, 1977</A>

(obviously this is a bad link)

and this one that was later imported OK:

<DT><A HREF="http://ancientaliens.bigcartel.com/" ADD_DATE="1451915382" PRIVATE="0" TOREAD="0" TAGS="embroidered,patch,space">Home / Ancient Aliens Patches</A>

http://ancientaliens.bigcartel.com

[gingerbeardman]

Importing a file containing only the 4 above gives:

1 bookmarks were successfully imported.

3 bookmarks could not be imported. Please check the logs for more details.

Linkding classes the top three, shown above, as invalid URLs.

For top top two it's the underscore that is the problem.

[sissbruecker]

Bit confusing to follow all of that, so the problem is solved now apart from the validation of the bookmark URLs?

[gingerbeardman]

Well, there are error messages but they seem to not stop the import. It will proceed with import.

Solved? I guess, but there is room for improvement of user experience.

[sissbruecker]

I think the issue here was using the app while the import was running, plus the import errors around URL validation. URL validation is covered by another ticket, and the import has become a lot faster which makes it less likely for the user to click around. Closing.