bookmarks
bookmarks copied to clipboard
Avoid crawling bookmark files responding HTTP error
The crawler currently only skipping bookmark links that responded with HTTP 404 status during the cron job. An URL returning a 500 status and a text/plain content type will result in an empty txt file being created in the bookmarks folder, such as:
< HTTP/1.1 500 Internal Server Error
< Server: [HIDDEN]
< Date: Tue, 23 Aug 2022 02:05:27 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
This PR marks a bookmark URL with any 4xx and 5xx errors as unavailable to stop this behavior.