bookmarks icon indicating copy to clipboard operation
bookmarks copied to clipboard

Avoid crawling bookmark files responding HTTP error

Open fengkaijia opened this issue 3 years ago • 0 comments

The crawler currently only skipping bookmark links that responded with HTTP 404 status during the cron job. An URL returning a 500 status and a text/plain content type will result in an empty txt file being created in the bookmarks folder, such as:

< HTTP/1.1 500 Internal Server Error
< Server: [HIDDEN]
< Date: Tue, 23 Aug 2022 02:05:27 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8

This PR marks a bookmark URL with any 4xx and 5xx errors as unavailable to stop this behavior.

fengkaijia avatar Aug 23 '22 02:08 fengkaijia